Re: [Boost-users] Threading and Asio

Subject: Re: [Boost-users] Threading and Asio
From: Anthony Foiani (tkil_at_[hidden])
Date: 2012-11-11 20:03:08

Gonzalo Garramuno <ggarra13_at_[hidden]> writes:

> I started playing with asio server/client model and I run into a
> problem.  I have a thread for the server, another for the client and
> another for the send/receive message.  This results in:
>
> write: Socket operation on non-socket
>
> as the send thread is not made aware of the socket initialized in the
> client thread.
> I'm wondering what is the solution to have the send in a separate thread.

Well, the brute-force answer is that you have separate state
signalling (with proper protection, either mutex + condition vars, or
atomics + sleeps).

I'm a little curious why you have a third thread for send/receive.
Are you doing synchronous read/writes?  Or are you doing some sort of
extra work (authentication, encryption, etc)?

In the "normal" case, your client is waiting to hear back from an
async_read call; in the read_handler, it does work, possibly sends
data (via async_write), then waits for more data (by calling
async_read "on itself").

Anyway, if you do want to use three threads, I believe that you need
to create three distinct io_service objects, and call
io_service::run() in each of those threads.  At that point, you can
use the typical ASIO calls (async_read etc) if you want to do i/o
directly; if you want to defer the work to a separate thread, you can
instead use "post" to put arbitrary work onto another io_service.

You can use thread pools by simply doing io_service::run() in multiple
threads, although you'll have to make sure that actions on behalf of a
given client/connection are done in sequence -- especially writes.
You can use asio strands for that; e.g.:

  http://stackoverflow.com/a/7756894/784478

Here's a sketch.  My apologies for conflating threads, classes, and
instances; also, my habit is to use Caps for classes, and the same
word in lower-case for instances.  I probably also got the order of
parameters to various functions wrong...

Finally, you'll probably want to be dealing with shared_ptr to Clients
and buffers; they live as long as they're still in the work queue
and/or live in a function, and go away if you don't pass them onto
another function.

Globals:
  io_service server_io; // server thread
  io_service client_io; // client thread
  io_service comm_io;   // send/receive worker thread

Main process:

  main:
    create io_services
    add synthetic work to all io_services
    thread server_thread ( Server::init );
    thread client_thread ( Client::init );
    thread comm_thread   ( Comm::init );
    server_thread.join();
    client_thread.join();
    comm_thread.join();
    exit;

Server thread:

  init:
    await_new_connection
    server_io.run()

  await_new_connection:
    create new Client
    server_io.async_connect( &Server::handle_async_connect, conn )
  
  handle_async_connect( conn ):
    if done: finish, then return
    client_io.post( &Client::add_client, conn )
    await_new_connection

  finish:
    remove synthetic work against all io_service objects
    possibly / probably call "cancel()" on them

Client thread:

  init:
    client_io.run()

  add_client( client ):
    comm_io.post( &Comm::await_data_for, client );

  handle_data( in_buf ):
    process in_buf
    to send data:
      comm_io.post( &Comm::send_data_for, client, out_buf );
    to read data:
      comm_io.post( &Comm::await_data_for, client );

Comm thread:

  init:
    comm_io.run();

  await_data_for( client ):
    comm_io.async_read( &Comm::handle_async_read, client )

  handle_async_read( in_buf, client ):
    slow_input_work_on( in_buf );
    client_io.post( &Client::handle_data, client, in_buf );

  send_data_for( client, out_buf ):
    slow_output_work_on( out_buf );
    comm_io.async_write( out_buf );