[Boost-users] Concurrency and session termination in boost::asio

Subject: [Boost-users] Concurrency and session termination in boost::asio
From: Scott Gifford (sgifford_at_[hidden])
Date: 2008-11-05 14:39:47

Hello,

I'm experimenting with boost::asio as the basis for communications and
event handling for a network server.

The server will support many clients, and should scale well to
machines with multiple processors/cores, so I'd like to use thread
pools to execute requests.  I've done this by calling io_service.run()
from multiple threads, which seems to be the normal way to do it.

Each client session always has an asynchronous read operation waiting
for new commands.  If the server has a notification for a thread, it
will send it by starting an asynchronous write on that session.
Notifications can come at any time, not just in response to a request,
so it is common for a session to simultaneously have oustanding
asynchronous read and write requests.  Sessions are terminated when
the socket is closed or when the client sends a quit command.

I'm trying to make sure that my handling of session termination is
threadsafe.  It seems that the normal way to handle this is to detect
an error or end-of-file in an asynchronous event handler, then delete
the object to end the session.  Because my server is using a
threadpool and waiting for both read and write events, it is possible
for a read and write callback to be running concurrently on different
CPUs.  In that case, if one of those callbacks detects an error and
deletes the object, the other will continue running using an invalid
object.

In a simple echo server I wrote to learn about boost::asio, I can see
these invalid accesses when running under valgrind with a large number
of clients, so unless I have made an error in my server, this appears
to be a real problem.

I rewrote it to have write errors handled by closing the session,
which causes the read callback to run, notice the session has been
closed, and delete the object.  This seemed to solve the problem, but
it still seems there is a potential problem if the read callback
finishes deleting the object while the write callback is running.

Is there a better way to handle session termination?

Can I use io_service::strand to prevent this by having the readers and
writers for the same session running in the same strand?

Thanks for any thoughts!

----Scott.