- Boost - lists.boost.org

asio queue OOM stress test
by christopher baus 16 Dec '05

16 Dec '05

I wrote a little program to stress the queue under low memory situations, and to demonstrate that there is no upper bound on memory usage when using async function calls. The first time I ran it under Linux the kernel killed mysqld and pretty much hung the system. The second time the kernel killed the test program. In neither run could I recover from the OOM error, which is one reason I wrote this. I want to demonstrate that under Linux you often can't catch exceptions from new, because your program is dead before the exception is thrown. Some might consider that a limitation of over committing memory managers like Linux and FreeBSD's, but that's how it works, and I think we are stuck with it. What is really interesting is on Windows, as I expected, I did get the exception and started running the handlers, but the program deadlocked each time I ran it after running about 600k function calls. I'm starting to wonder if this a bug unrelated to the OOM situation. #include <iostream> #include <boost/bind.hpp> #include <boost/asio.hpp> void doit(unsigned long* count) { ++(*count); if(!(*count % 100000)){ std::cout<<*count<<std::endl; } } int main(int argc, char* argv[]) { boost::asio::demuxer d; unsigned long call_count = 0; unsigned long post_count = 0; // // post messages until we run out memory and then // run them. try{ std::cout<<"posting events..."<<std::endl; for(post_count = 1;;++post_count){ // // Internally allocates queueing structures, and // a copy of the functor returned by bind. d.post(boost::bind(doit, &call_count)); if(!(post_count % 100000)){ // // print out something to show that // we are still alive. std::cout<<post_count<<std::endl; } } } catch(...){ // // Let's not do anything here in case we throw again. } // // There is a reasonable chance we might throw here, // because we are pretty much out of memory. Might // want to comment this line out. std::cout<<"caught exception. post_count: " <<post_count<<std::endl; try{ d.run(); } catch(...){ std::cout<<"caught exception running events."<<std::endl; // // Can't really do anything graceful here like continue // to handle connected sockets. // // I could try to call run again, but I doubt that's safe. // as we could have been mucking with internal structures // when the exception let go. } std::cout<<"all done"<<std::endl; std::cout<<"post_count: "<<post_count <<std::endl; std::cout<<"call_count: " <<call_count<<std::endl; return 0; }

3 10

Re: [boost] [serialization][bug]
by Oleg Abrosimov 16 Dec '05

16 Dec '05

It is really strange... I'm using boost 1.33.1 the compiler is MS VC 7.1.3088 In the file attached you can find VC solution and project (with modified demo file) that doesn't compile for me. Please, try to compile it. And if anyone else can do it under VC 7.1, please try it too. Thank you in advance, Oleg Abrosimov //------------------------------------- This was an issue with 1_33_0 that I believe was resolved with 1_33_1. I just tested this and it compiles with my VC 7.1 compiler with no errors. Looking at your code it seems that you're using 1_33_1 so I can't explain why we might be getting different results. Robert Ramey

2 1

[test] self-registration facilities
by Gennadiy Rozental 16 Dec '05

16 Dec '05

Hi, This message should serve as a temporary source of information regarding self registration faculties of Boost.Test. I will include the complete docs in next update. Here is the list of all available macros: ---------------------------------------------------- BOOST_AUTO_TEST_CASE( test_name ) ---------------------------------------------------- Simple free function like test case. Use like this: BOOST_AUTO_TEST_CASE( test ) { // test body } ---------------------------------------------------- BOOST_FIXTURE_TEST_CASE( test_name, F ) ---------------------------------------------------- Simple free function like test case. Employs struct F as a fixture. Use like this: struct F { F() { /* setup here */} ~F() { /* teardown here */} int i; }; BOOST_FIXTURE_TEST_CASE( test, F ) { // test body // i could be referenced here } ---------------------------------------------------- BOOST_AUTO_TEST_CASE_EXPECTED_FAILURES( test_name, n ) ---------------------------------------------------- Use this to specify number of expected failures in test case. Use like this BOOST_AUTO_TEST_CASE_EXPECTED_FAILURES( test1, 1 ) BOOST_AUTO_TEST_CASE( test1 ) { BOOST_CHECK( 2 == 1 ); } ---------------------------------------------------- BOOST_AUTO_TEST_CASE_TEMPLATE( test_name, type_name, type_list ) ---------------------------------------------------- Free function test case template for generating a test case for each type in type list. Use like this: typedef boost::mpl::list<char,int,float,double> test_types; BOOST_AUTO_TEST_CASE_TEMPLATE( test, T, test_types ) { // Body here; T is type under test } ---------------------------------------------------- BOOST_AUTO_TEST_SUITE( suite_name ) BOOST_AUTO_TEST_SUITE_END() ---------------------------------------------------- By default all test cases are registered in master test suite. There macros are used for constructing hierarchical test tree like this: BOOST_AUTO_TEST_SUITE( suite1 ); // this test case belongs to suite1 test suite BOOST_AUTO_TEST_CASE( test1 ) { // body here } BOOST_AUTO_TEST_SUITE_END(); ---------------------------------------------------- BOOST_FIXTURE_TEST_SUITE( suite_name, F ) ---------------------------------------------------- Similar to the BOOST_AUTO_TEST_SUITE, but force all the test cases within this test suite to use struct F as a fixture. BOOST_FIXTURE_TEST_SUITE( s, F ) // this test case use F as a fixture BOOST_AUTO_TEST_CASE( test1 ) { // body here } BOOST_AUTO_TEST_SUITE_END() ---------------------------------------------------- BOOST_GLOBAL_FIXTURE( F ) ---------------------------------------------------- Facilitate an ability to do global setup/teardown. Setup is invoked before any testing is started. Teardown - when all is done. Use like this: struct MyConfig { MyConfig() { std::cout << "global setup\n"; } ~MyConfig() { std::cout << "global teardown\n"; } }; BOOST_GLOBAL_FIXTURE( MyConfig ) You may have as many global fixtures as you want. To automatically setup init function define BOOST_TEST_MAIN in you main test module (in only one module in multi module test program) Alternatively you could define it explicitly. For example you could use it to rename a master test suite. Regards, Gennadiy

2 6

boost infrastructure questions
by Stefan Seefeld 16 Dec '05

16 Dec '05

For quite a while people have been discussing various enhancements to the boost infrastructure. What are these projects up to ? Notably: * Are there still plans to move to subversion ? If so, what is missing ? * What is the status of the move to boost.build version 2 ? * Are there still plans to 'officially' use buildbot to coordinate builds, test runs, and checkins ? I find it quite hard to interpret the regression test matrix since it is't obvious which test runs are in really in sync with the latest revision (or any revision, for that matter), and because there doesn't appear to be an obvious way to distinguish real test failures from failed prerequisites (incl. configuration errors, unsufficient resources, etc.) Thanks, Stefan

6 7

Re: [boost] asio formal review begins
by christopher baus 16 Dec '05

16 Dec '05

> What I did was to have the main thread accept incomming requests, and put > them in the queue, whereas a number of worker threads was taking these > requests from the queue, and execute them. I don't see how this model can > be absolutely inferior in all possible contexts. > I think your argument regarding the sync interface *might* be valid for other reasons, but I want to point out that this architecture only scales if you have an async front end that reads and writes from the socket and parses the connection into tasks in the main thread, and then handles these tasks synchronously in a thread pool. Say for instance a web server which handles HTTP request pipelines asynchronously and database connections synchronously. If the system contains long lived connections, then you are basically only servicing the number of connections in your thread pool, and there is a good chance the other connections will timeout. I do agree with Peter. If asio leads developers away from thread/connection architectures, then it has succeeded. As with a lot of technologies that are shiny and new, I think the industry has put too much emphasis on threads as the solution to all concurrency problems. I think we understand the problems better now. Christopher

4 4

asio review: are we still on topic?
by Arkadiy Vertleyb 16 Dec '05

16 Dec '05

Guys, This general discussion about which approach to networking is superior is very interesting (and useful for me personally), but I think we moved off topic. I think Chris (the author of asio) agreed that both approaches are valid, and claimed he gives both of them appropriate attention. So now, since I pointed to what I believe is a problem with the synchronous interface in asio (socket dependency upon the demuxer), doesn't the critique of synchronous approach sounds a little bit like: "It is inferior anyway, so it's a goot thing that it's not clean" ? Because nobody yet tried to convince me that socket dependency upon demuxer (even if for synchronous IO only) is a good thing. Regards, Arkadiy

5 4

[filesystem] Additional Decomposition Request
by me22 16 Dec '05

16 Dec '05

Hi all, The discussion a little bit ago regarding using path's grammar in a custom virtual filesystem reminded me of my experience when I made a simple unpacking program using boost::filesystem::path. One thing that wasn't as trivial as I might have liked was descending recursively down the internal hierarchy. It would have been nice to have head() and tail() decomposition functions ( though with possibly better names ) analogous to the branch_path() and leaf() functions, but for the front of the path. That way I could have simply looked for the head() element at the current level and recursed with tail(). ( Though I ended up doing it by having my get_file function delegate to one that worked with an iterator range, which is more efficient, but... ) The implementation is, of course, fairly trivial, something along the lines of the following: std::string path::head() { if ( empty() ) return std::string(); return *begin(); } path path::tail() { if ( empty() ) return path(); return std::accumulate( next(begin()), end(), std::divides<path>() ); } And the obvious has_head() and has_tail(). The usage would be something similar to: file &directory::get_file(boost::filesystem::path p) { if ( !p.has_tail() ) return files[ p.head() ]; return subdirectories[ p.head() ].get_file( p.tail() ); } Assuming appropriate ( not part of boost ) directory and file classes. Opinions? Interest? - Scott McMurray P.S. I'm particularly fond of the accumulate/divides hack :P

1 0

[serialization][bug]
by Oleg Abrosimov 16 Dec '05

16 Dec '05

Hello! Some days ago I've faced with very strange problem with boost::serialization it doesn't compile, though docs says that it should. After some research I've found that changing the order of header files included fixes the problem. It can be illustrated on a base of the demo.cpp program included in distribution: The original order: #include <boost/archive/text_iarchive.hpp> #include <boost/archive/text_oarchive.hpp> #include <boost/serialization/base_object.hpp> #include <boost/serialization/utility.hpp> #include <boost/serialization/list.hpp> The modified order [doesn't compile!]: #include <boost/serialization/base_object.hpp> #include <boost/serialization/utility.hpp> #include <boost/serialization/list.hpp> #include <boost/archive/text_iarchive.hpp> #include <boost/archive/text_oarchive.hpp> It is crucial to include these files first #include <boost/archive/text_iarchive.hpp> #include <boost/archive/text_oarchive.hpp> it is because of many files (may be indirectly) include boost/serialization/type_info_implementation.hpp header that uses BOOST_SERIALIZATION_DEFAULT_TYPE_INFO macro without defining it, and #include <boost/archive/text_iarchive.hpp> #include <boost/archive/text_oarchive.hpp> headers define this macro in some way. to conclude: I have found it very disappointing that changing of order of included headers changes code correctness so dramatically. I want it to be fixed in some way. Thanks for great and incredibly helpful library anyway, Oleg Abrosimov. PS: The compiler (VC7.1) output: Compiling... demo.cpp D:\home\olegabr\devel\boost_1_33_1\boost\archive\detail\oserializer.hpp(131) : error C2027: use of undefined type 'boost::serialization::extended_type_info_null<T>' with [ T=bus_schedule ] D:\home\olegabr\devel\boost_1_33_1\boost\archive\detail\oserializer.hpp(131) : see reference to class template instantiation 'boost::serialization::extended_type_info_null<T>' being compiled with [ T=bus_schedule ] D:\home\olegabr\devel\boost_1_33_1\boost\archive\detail\oserializer.hpp(128) : while compiling class-template member function 'bool boost::archive::detail::oserializer<Archive,T>::is_polymorphic(void) const' with [ Archive=boost::archive::text_oarchive, T=bus_schedule ] D:\home\olegabr\devel\boost_1_33_1\boost\archive\detail\oserializer.hpp(264) : see reference to class template instantiation 'boost::archive::detail::oserializer<Archive,T>' being compiled with [ Archive=boost::archive::text_oarchive, T=bus_schedule ] D:\home\olegabr\devel\boost_1_33_1\boost\archive\detail\oserializer.hpp(263) : while compiling class-template member function 'void boost::archive::detail::save_non_pointer_type<Archive,T>::save_standard::invoke(Archive &,const bus_schedule &)' with [ Archive=boost::archive::text_oarchive, T=bus_schedule ] D:\home\olegabr\devel\boost_1_33_1\boost\archive\detail\oserializer.hpp(322) : see reference to class template instantiation 'boost::archive::detail::save_non_pointer_type<Archive,T>::save_standard' being compiled with [ Archive=boost::archive::text_oarchive, T=bus_schedule ] D:\home\olegabr\devel\boost_1_33_1\boost\archive\detail\oserializer.hpp(310) : while compiling class-template member function 'void boost::archive::detail::save_non_pointer_type<Archive,T>::invoke(Archive &,const bus_schedule &)' with [ Archive=boost::archive::text_oarchive, T=bus_schedule ] D:\home\olegabr\devel\boost_1_33_1\boost\archive\detail\oserializer.hpp(536) : see reference to class template instantiation 'boost::archive::detail::save_non_pointer_type<Archive,T>' being compiled with [ Archive=boost::archive::text_oarchive, T=bus_schedule ] D:\home\olegabr\devel\boost_1_33_1\boost\archive\basic_text_oarchive.hpp(78) : see reference to function template instantiation 'void boost::archive::save<Archive,T>(Archive &,T &)' being compiled with [ Archive=boost::archive::text_oarchive, T=const bus_schedule ] D:\home\olegabr\devel\boost_1_33_1\boost\archive\detail\interface_oarchive.hpp(78) : see reference to function template instantiation 'void boost::archive::basic_text_oarchive<Archive>::save_override<T>(T &,int)' being compiled with [ Archive=boost::archive::text_oarchive, T=const bus_schedule ] demo.cpp(286) : see reference to function template instantiation 'Archive &boost::archive::detail::interface_oarchive<Archive>::operator <<<const bus_schedule>(T &)' being compiled with [ Archive=boost::archive::text_oarchive, T=const bus_schedule ] D:\home\olegabr\devel\boost_1_33_1\boost\archive\detail\oserializer.hpp(131) : error C2146: syntax error : missing ';' before identifier 'typex' D:\home\olegabr\devel\boost_1_33_1\boost\archive\detail\oserializer.hpp(131) : error C2065: 'typex' : undeclared identifier

2 1

Re: [boost] asio formal review begins
by christopher baus 16 Dec '05

16 Dec '05

> I happen to > think that the fact many people have been trained to think about sockets synchronously is most definitely for the worse. Totally agree >> I just want to point out that async reading and writing could >> be a function of an async I/O demuxer and not the socket >> itself. The socket could be passed to the demuxer and not >> vice versa. > The above design is something I can fundamentally disagree with > ;) That sort of design, in my opinion, relegates asynchronous > operations to "second class citizens". I'm not religious about this, but it could be argued that it would put sync and async on the same level. For instance you could consider the POSIX interface to be os::sync_io::write(fd, buffer, bytes) and the os::sync_io is just left out. Then my interface would be os::async_demuxer::write(fd, buffer, bytes); But honestly this isn't something that concerns me. > There is a relatively straightforward mapping between one and > the other. In the future I want to exploit this mapping further > by investigating the use of expression templates (perhaps > similar to Boost.Lambda) to permit the encoding of a sequence of > operations in a synchronous programming style. The operations > could then be executed either synchronously or asynchronously as needed. Not to take away from asio (which I think is great BTW), but before jumping into discussion about expression templates, I want to point out that accepting handlers as template parameters and passing them by value isn't free in asio. The end result is a dynamic copy of the handler per operation in a system with potentially thousands of operations/second. This is going to make potential pooling inefficient because the library doesn't know what size and how many handlers are used until run time (what happens if one large handler is queued along with a bunch of small ones?). The handler functions are called via a function pointer, so there isn't any performance advantage by passing handler functors by value, and the deferred nature of async i/o makes compile time binding impossible (ok I shouldn't say impossible with this group, but extremely unlikely). I'm going to discuss this more in my review, but there is a significant amount of overhead that is an artifact of the value semantics. I think a parallel interface which accepts handlers by reference (or pointer) should be considered. The result would be that the user could specify the same handler instance for multiple operations, and asio could pool fixed sized queue nodes. I suspect that in applications handling a large number of connections the result would be improved reliability, performance, and in some cases lower memory usage. Anybody that's interested in the *nix implementation of this should look at detail/reactor_op_queue.hpp, which is an interesting read. The allocation I'm discussing happens in enqueue_operation(). Similar allocations happen in win_iocp_socket_service.hpp and win_iocp_demuxer_service. I think the allocation for value functors could be done in a cross platform area of the library, and a concrete handler object pointer could be passed to platform specific implementations which allocate fixed sized structures from pools. Other than that concern with memory management, I become more impressed by the library the more time I spend with it. It is really a great piece of work. Cheers, Christopher

1 0

[filesystem] problem with filename length (bug?)
by Allmen, Michael-P57512 16 Dec '05

16 Dec '05

Here is the code I am using (the problem also occurs with the simple_ls.cpp sample)... #include <iostream> #include "boost/filesystem/operations.hpp" #include "boost/filesystem/path.hpp" namespace fs = boost::filesystem; int main( int argc, char* argv[] ) { fs::path dir( "somedir" ); fs::directory_iterator end; for( fs::directory_iterator i( dir ); i != end; ++i ) { std::cout << dir->leaf() << std::endl; } return 0; } The problem is that once the program finds a file at least 22 characters long, it will not print any filenames longer than first file correctly. In "somedir" are several long filenames. Here is the contents of the directory in the order the files were created: somereallyreallylongfilename somereallylongfilename someotherreallylongfilename somereallyreallylongfilenam3 somereallyreallylongfilenames The program outputs: somereallyreallylongfilename somereallylongfilename someotherreallylongfilename somereallyreallylongfilenam3 somereallyreallylongfi (here the output stops and goes back to my prompt) Anyone else had this problem? Thanks, Mike

2 1