Re: [boost] [asio] Active Objects Library

From: Christopher Kohlhoff (chris_at_[hidden])
Date: 2006-10-24 17:04:47

Hi Scott,

Scott <cheesy4poofs_at_[hidden]> wrote:
> I recently switched to a pool of worker threads using a single
> threaded io_service.  If you remember, I posted several times
> about SSL wasn't working reliably.  Well, the single threaded
> io_service seems to have fixed that problem (I do believe the
> SSL portion of ASIO has thread issues).
>
> However, I could never figure out how to avoid thread locking.
> The documentation says that asio::buffer() does not make a
> copy of the data, instead it makes copies of the pointer to
> the buffer.  Because of this, I had to copy my heap data to be
> sent over the socket into a deque before calling post().

I think this buffer stuff is orthogonal to the issue of locking,
but have you seen the reference counted buffer example?

<http://asio.sourceforge.net/boost_asio_0_3_7/libs/asio/doc/examples/buffers_reference_counted_cpp.html>

> After the handler is called (the socket write has finished), I
> then remove the first entry in the deque and submit the next
> entry.  This requires thread locking on the deque.

Ok, so here...

> I should also point out that because of our design, 2 or more
> worker threads could be doing work that will require data to
> be sent out over the same socket at exactly the same time.  I
> had to lock access to the socket since the docs say it isn't
> thread safe.

...and here are some places where you are doing some locking. An
explicit lock will prevent concurrent access to the data, yes,
but another approach is to keep all accesses to the data from
the same thread. This second approach is what this whole active
object thread is about.

You say your design now has a single thread calling
io_service::run(), so let's specify that accesses to the deque
and socket are *only* allowed to occur from that thread. Problem
solved :)

Seriously though, to implement this solution you need to use
io_service::post() in conjunction with boost::bind(). These
calls let you package up and post units of work to be executed
by the io_service. Since only one thread is calling
io_service::run(), all of these work units will be executed by
that thread.

For example, your connection class might look something like
this:

class connection
{
public:
  connection(io_service& i)
    : io_service_(i), socket_(i)
  {
  }

  ... connection establishment stuff here ...

  void write_message(const std::string& msg)
  {
    io_service_.post(
        boost::bind(
          &connection::do_write_message,
          this, msg));
  }

private:
  void do_write_message(const std::string& msg)
  {
    if (msgs_.empty())
    {
      // No write in progress, so start write now.
      msgs_.push_back(msg);
      async_write(socket_,
          buffer(msgs_.front()),
          boost::bind(
            &connection::handle_write,
            this, _1);
    }
    else
    {
      // Another write already in progress.
      msgs_.push_back(msg);
    }
  }

  void handle_write(error e)
  {
    if (!e)
    {
      msgs_.pop_front();
      if (!msgs_.empty())
      {
        // Another message is ready to write.
        async_write(socket_,
            buffer(msgs_.front()),
            boost::bind(
              &connection::handle_write,
              this, _1);
      }
    }
  }

  io_service& io_service_;
  tcp::socket socket_;
  std::deque<std::string> msgs_;
};

The private member functions do_write_message() and
handle_write() are the only places that access the socket and
deque. These functions are only ever used as callback handlers
with io_service::post() or async_write(), which means that they
are guaranteed to be called only from the thread that is calling
io_service::run().

The public member function write_message() is written so that it
can be called from any thread at any time (since the
io_service::post() function is designed that way). It doesn't
matter if "2 or more worker threads" need to call
connection::write_message() since it's now perfectly safe to do
so.

No locks in sight :)

Cheers,
Chris