March 2005 - Boost - lists.boost.org

Re: Asynchronous I/O
by Don G 29 Mar '05

29 Mar '05

Hi Jon, Thanks for the reply! Please see below for my responses. --- Jonathan Wakely <cow(a)compsoc.man.ac.uk> wrote: > How the kernel chooses to implement the aio system calls is an > implementation detail - it certainly doesn't require use of > Boost.Thread types! I think what Robert was saying is that an > application can use POSIX aio without ever having to use or > know about POSIX threads. But if it calls your callback on a thread other than main, you do care and need to know that detail. If the callback is called on the main thread, that thread either has to cooperate or for signals (again) be prepared to handle the special execution state inside the callback. > You and Don seem to be requiring that the application > explicitly create and manage threads. Several people have > suggested they'd like to use some form of AIO without having > to use threads. In the "Asynchronisity" thread, I suggest a mechanism that has certain thread-safe properties, but the user need never create a thread. That being said, I am proposing the the network layer (not the application) create and manage I/O threads. The callbacks it makes to the application would occur in those threads, but by using a technique like bind_async_call (see the "Asynchronicity" thread for details), the callback to user code can be delivered on the one and only thread that the user knows about. In pseudo code: char buf[256]; channel ch; stream::ptr st; int main () { nexus nx; ch.open(&nx); network::ptr net = ?; // create the network object st = net->new_address("foo:1234")->new_stream(); st->async_connect(ch.bind_async_call(&on_connect)); nx.pump(); // callbacks go until nx.close() is called } void on_connect () { st->async_read(buf, sizeof buf, ch.bind_async_call(&on_read_done)); } void on_read_done () { ... } Notice that all this is done inside main(). The user, however, must cooperate to deliver the async messages (the nx.pump() call). But this level of cooperation is required for most async mechanisms on most systems. I hope this clarifies what I am proposing. Best, Don __________________________________ Do you Yahoo!? Yahoo! Small Business - Try our new resources site! http://smallbusiness.yahoo.com/resources/

1 0

Asynchronous I/O
by Boris 29 Mar '05

29 Mar '05

While trying to find a reasonable interface for a socket class to support asynchronous I/O I was examining various implementations. Scott Woods had thought loudly about an asynchronous library which could be the platform for a network library and maybe others. I like this idea but look at this mess: 1) Signal-Driven I/O SIGIO is sent to a process whenever something happens with the socket. As there are too many things happening with a TCP socket the signal is generated too often and "signal-driven I/O is next to useless with a TCP socket" (W. Richard Stevens in his network bible). As there is just one signal handler per signal it shouldn't be used in a library anyway. 2) Threads There are different solutions possible eg. every thread calling a blocking function or one thread blocking while the other threads handle requests. However these are just variations of the blocking I/O model and multiplexing with the disadvantage of increased complexity because of threads. 3) AIO AIO is part of the Single UNIX Specification and an official standard for asynchronous input and output. The AIO API is based on file descriptors and defines a few functions inluding aio_read() and aio_write(). I/O is asynchronous because the process is notified when an input or output operation is completed - there is either a signal sent to the process or a callback function is executed (as a thread). Further more it is possible to poll a file descriptor if the process doesn't want to wait for the signal or the callback function. Now to the disadvantages: Linux 2.6 doesn't support AIO with sockets (see http://lse.sourceforge.net/io/aio.html) There are some AIO patches for Linux 2.4 but nothing official. Even if AIO support was complete there are no aio_connect() and aio_accept() functions. I wonder if connections have to be built up synchronously (I guess so as AIO wasn't designed only for sockets anyway). 4) I/O completion ports This I/O model is based on a concept called port. A port is similar to a multiplexor and waits for several input or output operations to complete. There can be several ports each multiplexing on several file descriptors. Windows XP, Windows 2000 Pro and Windows NT 3.5 support I/O completion ports. However the application has to call a blocking function for a port to be notified when an I/O operation is complete. 4) Event completion framework The event completion framework behaves like I/O completion ports. However the process can be notified of different types of events not only of the completion of I/O operations. Solaris 10 supports these events: AIO, timer, file descriptor, user-defined and waking up threads (see http://developers.sun.com/solaris/articles/event_completion.html) FreeBSD, OpenBSD and NetBSD support: read, write, AIO, file descriptor, process IDs (exit, fork etc. of other processes), signal, timer and network (see http://www.freebsd.org/cgi/man.cgi?query=kqueue&apropos=0&sektion=0…) The framework of *BSD is called kqueue and not compatible to Solaris. I don't know what other operating systems like Mac OS X support. There is kqueue() is their man pages (see http://developer.apple.com/documentation/Darwin/Reference/ManPages/index.ht…) but I don't know if it is supported. AIO doesn't seem to be supported. Microsoft operating systems provide WaitForMultipleObjects() since Windows 95 which supports these objects: change notification, console input, event, job, memory resource notification, mutex, process, semaphore, thread and waitable timer (see http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dllproc/ba…) Then I had a look at ACE as I wondered how they support asynchronous I/O with their Proactor class on different operating systems: They don't. They use this preprocessor directive in the beginning of Proactor.h: #if ((defined (ACE_WIN32) && !defined (ACE_HAS_WINCE)) || (defined (ACE_HAS_AIO_CALLS))) // This only works on Win32 platforms and on Unix platforms supporting POSIX aio calls. Someone wanted to port a Windows applications to Linux which was using the Proactor class. The ACE people told him he should say good-bye to Proactor (see http://groups.yahoo.com/group/ace-users/message/38548) Last but not least Linux 2.5 introduced a new API called epoll (see http://www.linuxmanpages.com/man4/epoll.4.php) It doesn't support asynchronous I/O but tries to get rid of all the disadvantages select() and poll() have. The Linux world seems to pursue the traditional multiplexing approach trying to improve it. Now who has an idea how the asynchronous C++ library should look like? ;) Boris

9 12

Asynchronicity (long)
by Don G 29 Mar '05

29 Mar '05

Hi All, I've renamed the subject from "Asynchronous I/O" because I wanted to float the notion that async behavior not be viewed as an aspect of I/O, though it is highly desired in that space. Below is a sketch and some pseudo-code of how I've done this over the past few years, in a portable manner. The trick to portability for me was integrating into various kinds of threads or main loops. For example: the main GUI thread of a Windows application, or a Motif (Xt) based application, or a Mac application's main event loop. These are the hard ones. Once they are possible, most (more likely "all") others are possible as well. Unfortunately, the needs of these integrators is not portable, but can be hidden behind simple wrapper classes for anticipated cases. To handle other cases, the integration mechanism needs to be visible. In the case of the GUI thread integrations, I have a single class to instantiate, though its details were wildly different for each platform. Depending on how this discussion goes, I plan to convert these ideas into a proposed library. Best, Don ------ Sketch -------- 1. The user creates one or more "nexus" objects. These objects store the queue of events, typically one-per-thread. Since some action is typically required as messages are enqueued, a callback is supplied to open() which gets called as each message is added. 2. The owner of the nexus is responsible for calling the pump() method to drain the messages. This method is intended for only one thread and hence is not internally thread-safe (i.e., BYOM ... Bring Your Own Mutex). 3. To queue calls, client code needs to create a "channel" object and connect it with a nexus. The reason for this object is that it is a proxy for the "ability to still dispatch a message". In other words, if a receiver object finds itself in its destructor, it will want to cancel any queued calls to it. Of course, this can be applied externally to objects, which is why channel is not a base class. 4. Once a channel is open, the async_call() method is used to put a message in flight. This is thread-safe. 5. Where other code needs a function<>, the channel can be used to create a shim function<> that accepts N arguments, and binds them into a queued call. ------ Pseudo-code -------- class nexus { friend class channel; public: typedef function<void (void)> message; // Return true if there is an object associated with // the current thread. static bool available (); // Return the object associated with the current // thread. Throw if there is none. static nexus & get_current (); // Connect this object with the current thread. The // ntfy callback is made whenever a message is added // to this queue. void open (const message & ntfy); void close (); // Remove a message from the queue. Don't wait any // longer than "t". If t == 0 and there is a message // in the queue, dispatch it now. void pump (const timeout & t = timeout::infinite); private: struct entry { channel * ch; message msg; }; typedef std::list<entry> messages; messages msgs_; message ntfy_; void enqueue (channel * ch, const message & msg) { msgs_.push_back(entry(ch, msg)); ntfy_(); } // Remove all entry's in msgs_ with == channel*. void purge (channel * ch); }; class channel { typedef nexus::message message; public: void open (nexus * nx = 0) { if (!nx) nx = &nexus::get_current(); close(); target_ = nx; } void close () { target_->purge(this); } void async_call (const message & msg) { target_->enqueue(this, msg); } // Create a callback that calls "this->async_call(msg)". message bind_async_call (const message & msg); // Create a callback that accepts T and calls // "this->async_call(bind(msg, t))". template <typename T1> function<void (T)> bind_async_call (const function<void (T)> & msg); // More flavors of bind_async_call() private: nexus * target_; }; __________________________________ Do you Yahoo!? Yahoo! Small Business - Try our new resources site! http://smallbusiness.yahoo.com/resources/

2 1

ublas test4+5 failures
by Stefan Slapeta 29 Mar '05

29 Mar '05

Please could somebody look into these: http://tinyurl.com/49lg5 http://tinyurl.com/4lvn8 These failures have occured for a long time now. The failing assertions produce popups that are very annoying for tests without the build monitor! Thanks, Stefan

1 0

type_traits: recent changes break EDG compilations
by Ralf W. Grosse-Kunstleve 29 Mar '05

29 Mar '05

The current Boost CVS does not work for compiling Boost.Python with the EDG front-end versions 238, 245, and 303 (and possibly any other EDG). E.g. using icc 8.1: icc -fPIC -DBOOST_DISABLE_THREADS -DNDEBUG -O2 -DBOOST_PYTHON_MAX_BASES=2 -I/net/legless/scratch1/rwgk/hot/boost -I/usr/local_cci/Python-2.4_ticker/include/python2.4 -c -o boost/libs/python/test/args.os /net/legless/scratch1/rwgk/hot/boost/libs/python/test/args.cpp /net/legless/scratch1/rwgk/hot/boost/boost/python/detail/invoke.hpp(75): error: expression must have (pointer-to-) function type return rc(f( BOOST_PP_ENUM_BINARY_PARAMS_Z(1, N, ac, () BOOST_PP_INTERCEPT) )); ^ detected during: The full error messages are here: http://cci.lbl.gov/~rwgk/tmp/edg238_problem_2005_03_27_1250 http://cci.lbl.gov/~rwgk/tmp/edg245_problem_2005_03_27_1250 http://cci.lbl.gov/~rwgk/tmp/edg303_problem_2005_03_27_1250 I've tracked down the failures to this CVS commit: date: 2005/03/23 11:22:25; author: johnmaddock; Modified function testing to make it configuration-friendly, future changes should require no more than a config macro change. If I check out the CVS HEAD, but run this command % cvs -q update -P -D'2005/03/23 11:00 UTC' U config.hpp U is_convertible.hpp U is_function.hpp U is_member_function_pointer.hpp U type_traits_test.hpp U detail/is_function_ptr_helper.hpp U detail/is_function_ptr_tester.hpp U detail/is_mem_fun_pointer_impl.hpp U detail/is_mem_fun_pointer_tester.hpp in boost/boost/type_traits, compilation succeeds again. The full diffs are here: http://cci.lbl.gov/~rwgk/tmp/edg_problem_2005_03_27_1250_type_traits_diffs I noticed that is_convertible.hpp was modified at a later time (2005/03/24 18:18:36), but this change does not make a difference for the EDG failures. Thanks in advance for any help! Cheers, Ralf __________________________________ Do you Yahoo!? Yahoo! Small Business - Try our new resources site! http://smallbusiness.yahoo.com/resources/

2 1

Threading and Visual C++
by Rasmus Christian Kaae 29 Mar '05

29 Mar '05

Hello, I am currently starting a project using the boost libraries and Visual C++. In this project we wish to minimize the number of DLL's shipped with our final product. However, we have discovered that the boost thread library is fixed to be dynamically loadable, is there any way to get around this? Short: How do i statically link the boost thread library in Visual C++ projects? -- Best regards / Med venlig hilsen Rasmus Christian Kaae - kaae(a)cs.auc.dk or kaae(a)daimi.au.dk Office: E4-119, Aalborg University. Phone: +45 75 88 04 13 Mobile: +45 28 72 04 13

2 1

preprocessor puzzle
by Edward Bishop 29 Mar '05

29 Mar '05

I came up with this great macro called IS_UNSIGNED_CHAR() such that IS_UNSIGNED_CHAR(unsigned char) expands to 1 and IS_UNSIGNED_CHAR(hello world) or any other input, expands to 0. It almost works, but it gets confused by IS_UNSIGNED_CHAR(unsigned world) which expands to garbage: BOOST_PP_IIF_I, EQUALS_42_57, RETURN_0)((42) world_U_I_O_P)); Any suggestions? Here is the code: #include <boost/preprocessor.hpp> #define Q_W_E_R_T_Y_unsigned 42) #define char_U_I_O_P (57 // Transmogrify t #define MOG(t) Q_W_E_R_T_Y_ ## t ## _U_I_O_P // Check if t is (42)(57) #define EQUALS_42_57(t) BOOST_PP_AND(BOOST_PP_EQUAL(42,BOOST_PP_SEQ_ELEM(0, t)), \ BOOST_PP_EQUAL(57,BOOST_PP_SEQ_ELEM(1, t)) ) // return 0 #define RETURN_0(t) 0 #define IS_UNSIGNED_CHAR(t) BOOST_PP_IIF(BOOST_PP_EQUAL(2,BOOST_PP_SEQ_SIZE((MOG(t)))), \ EQUALS_42_57, RETURN_0)((MOG(t))) // Here is some test data: IS_UNSIGNED_CHAR(unsigned char); // should expand to 1 IS_UNSIGNED_CHAR(hello world); // should expand to 0 IS_UNSIGNED_CHAR(unsigned world); // should expand to 0

4 10

[patch] lambda bug with GCC debug mode
by Jonathan Wakely 29 Mar '05

29 Mar '05

The libstdc++ debug mode doesn't work with forward declarations of std templates (since when using the debug mode std::vector is really __gnu_debug_def::vector, visible in std due to GCC's "strong using" extension) Lambda's boost/lambda/detail/operator_return_type_traits.hpp seems to account for this, but only checks for GCC 3.4 not 4.0, causing the error shown here for GCC 4.x in debug mode (*): http://www.redi.uklinux.net/src/boost/status/cs-Linux-links.html#lambda-con… The workaround on line 876 should be changed to account for GCC 4.x or alternatively, test for the debug mode explicitly. The attached patch does the test explicitly and disables all forward decls of std templates when _GLIBCXX_DEBUG is defined. I don't know whether it's preferred to simply disable the forward decls for non-debug mode as well. jon (*) regression test results for GCC 4.0 & 4.1 on x86_64 GNU/Linux here: http://www.redi.uklinux.net/src/boost/status/cs-Linux.html All toolsets use GCC's concept-checks, the gcc-debug-mode toolset also uses the libstdc++ debug mode. -- "Always Read the Label" - anon.

1 0

[mpl] eval_if has default parameters?
by Andy Little 29 Mar '05

29 Mar '05

The following compiles (VC7.1, gcc3.2), and I am wondering if this is the intended behaviour(boost 1.32) Note the two default parameters to eval_if. here #include <iostream> #include <boost/mpl/eval_if.hpp> int main() { typedef boost::mpl::eval_if< boost::mpl::true_ // whatever nullary metafunction >::type type; std::cout << type::value <<'\n'; // outputs 0 } regards Andy Little

1 1

Re: Serialization issuses under MS VC++ 7.1 ....
by Jason Earl 29 Mar '05

29 Mar '05

If you have look at my original code, it does use BOOST_SERIALIZATION_SPLIT_MEMBER() inside the class (sorry the code doesn't intend with tabs nicely). I also have tried including both... #include <boost/serialization/split_member.hpp> #include <boost/serialization/split_free.hpp> but neither work. I assume that the split free is obviously used by the list data type as it's not intrusive, and the member one is what I should include and have been including. I'll look running the tests later when I get round to figuring out how to run them etc. I'll need to RTFM first though. From the regression thingy, it said MSVC++7.1 worked OK though. I think this could be a little bug in the library (I won't speak too loudly here as I'm still a C++ beginner, and therefore there's a good chance I'm still doing something very dumb), maybe someone else could confirm they get the same problem as me when trying to serialize an class member varaible that is a list of objects using load / save methods that are class members under VS.NET 2003, like in my example.... anotherwords I have class A with separate load/save members, and then I have class B which has a member inside it that is of type std::list<A*>. Sorry in advance if I break the threaded reply structure on this list. I didn't have the list mail me replies. I'm new to all this mailing list stuff. Jason

1 0