[boost] Asynchronous I/O

From: Boris (boris_at_[hidden])
Date: 2005-03-17 10:08:07

While trying to find a reasonable interface for a socket class to support 
asynchronous I/O I was examining various implementations. Scott Woods had 
thought loudly about an asynchronous library which could be the platform for 
a network library and maybe others. I like this idea but look at this mess:

1) Signal-Driven I/O
SIGIO is sent to a process whenever something happens with the socket. As 
there are too many things happening with a TCP socket the signal is 
generated too often and "signal-driven I/O is next to useless with a TCP 
socket" (W. Richard Stevens in his network bible). As there is just one 
signal handler per signal it shouldn't be used in a library anyway.

2) Threads
There are different solutions possible eg. every thread calling a blocking 
function or one thread blocking while the other threads handle requests. 
However these are just variations of the blocking I/O model and multiplexing 
with the disadvantage of increased complexity because of threads.

3) AIO
AIO is part of the Single UNIX Specification and an official standard for 
asynchronous input and output. The AIO API is based on file descriptors and 
defines a few functions inluding aio_read() and aio_write(). I/O is 
asynchronous because the process is notified when an input or output 
operation is completed - there is either a signal sent to the process or a 
callback function is executed (as a thread). Further more it is possible to 
poll a file descriptor if the process doesn't want to wait for the signal or 
the callback function.
Now to the disadvantages: Linux 2.6 doesn't support AIO with sockets (see 
http://lse.sourceforge.net/io/aio.html). There are some AIO patches for 
Linux 2.4 but nothing official. Even if AIO support was complete there are 
no aio_connect() and aio_accept() functions. I wonder if connections have to 
be built up synchronously (I guess so as AIO wasn't designed only for 
sockets anyway).

4) I/O completion ports
This I/O model is based on a concept called port. A port is similar to a 
multiplexor and waits for several input or output operations to complete. 
There can be several ports each multiplexing on several file descriptors. 
Windows XP, Windows 2000 Pro and Windows NT 3.5 support I/O completion 
ports. However the application has to call a blocking function for a port to 
be notified when an I/O operation is complete.

4) Event completion framework
The event completion framework behaves like I/O completion ports. However 
the process can be notified of different types of events not only of the 
completion of I/O operations. Solaris 10 supports these events: AIO, timer, 
file descriptor, user-defined and waking up threads (see 
http://developers.sun.com/solaris/articles/event_completion.html). FreeBSD, 
OpenBSD and NetBSD support: read, write, AIO, file descriptor, process IDs 
(exit, fork etc. of other processes), signal, timer and network (see 
http://www.freebsd.org/cgi/man.cgi?query=kqueue&apropos=0&sektion=0&manpath=FreeBSD+5.0-current&format=html). 
The framework of *BSD is called kqueue and not compatible to Solaris.

I don't know what other operating systems like Mac OS X support. There is 
kqueue() is their man pages (see 
http://developer.apple.com/documentation/Darwin/Reference/ManPages/index.html) 
but I don't know if it is supported. AIO doesn't seem to be supported.

Microsoft operating systems provide WaitForMultipleObjects() since Windows 
95 which supports these objects: change notification, console input, event, 
job, memory resource notification, mutex, process, semaphore, thread and 
waitable timer (see 
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dllproc/base/waitformultipleobjects.asp).

Then I had a look at ACE as I wondered how they support asynchronous I/O 
with their Proactor class on different operating systems: They don't. They 
use this preprocessor directive in the beginning of Proactor.h:
#if ((defined (ACE_WIN32) && !defined (ACE_HAS_WINCE)) || (defined 
(ACE_HAS_AIO_CALLS)))
// This only works on Win32 platforms and on Unix platforms supporting POSIX 
aio calls.

Someone wanted to port a Windows applications to Linux which was using the 
Proactor class. The ACE people told him he should say good-bye to Proactor 
(see http://groups.yahoo.com/group/ace-users/message/38548).

Last but not least Linux 2.5 introduced a new API called epoll (see 
http://www.linuxmanpages.com/man4/epoll.4.php). It doesn't support 
asynchronous I/O but tries to get rid of all the disadvantages select() and 
poll() have. The Linux world seems to pursue the traditional multiplexing 
approach trying to improve it.

Now who has an idea how the asynchronous C++ library should look like? ;)

Boris