December 2005 - Boost - lists.boost.org

Boost.Python and Unicode
by Ben Artin 17 Dec '05

17 Dec '05

My C++ code uses std::string as storage for UTF-8 strings. I am exporting some of my C++ APIs to Python using Boost.Python, and I would like to somehow Boost.Python about the fact that my strings are UTF-8 strings. Ideally, this would mean that Boost.Python would convert my std::strings to Python unicode objects on the way from C++ to Python, and that any Python unicode objects passed into my C++ library would be converted to UTF-8 encoded std::strings. What's the best way to go about doing this? Thanks, Ben -- If this message helped you, consider buying an item from my wish list: <http://artins.org/ben/wishlist> I changed my name: <http://periodic-kingdom.org/People/NameChange.php>

2 1

Re: [boost] asio queue OOM stress test -- Windows deadlock
by christopher baus 16 Dec '05

16 Dec '05

> template <typename Allocator = std::allocator<void> > > class limits_service > { > ... > }; > > template <typename Service> > class basic_limits > { > public: > basic_limits(demuxer_type& d); > void post_queue(std::size_t value); > std::size_t post_queue() const; > ... and so on ... > }; > > typedef basic_limits<limits_service<> > limits; > > Usage: > > boost::asio::demuxer d; > boost::asio::limits l(d); > l.post_queue(42); > I thought about this some more, and I think it might be better to pass the limits to the demuxer constructor. You have to forgive me. I'm pretty anal about memory management. I tend to allocate my memory at startup on Linux servers because of the way the OOM-killer works. I'm working on a patch to reactor_op_queue.hpp to pool as many of your internal data structures as possible. If I had the limits at construction, I could allocate the necessary internal structures when the demuxer is instantiated. I think this the right thing to do for server applications because if the admin configures a server to handle n number of connections I want to make my best attempt to handle those connections. If the system doesn't have enough resources I think the admin should know about it at startup. This allocation strategy should probably be policy based, because for clients allocating from the heap is more appropriate. I wrote a bit more about this a while back here: http://baus.net/memory-management Thanks again, Christopher

2 2

[filesystem] i18n merge complete
by Beman Dawes 16 Dec '05

16 Dec '05

The Boost.Filesystem i18n branch has been merged into the main trunk HEAD in CVS. This is a major upgrade, several years in the making. It supports wide character filenames, and has a pile of other improvements. It implements the TR2 library proposal before the C++ Standards Committee. Most, but not all, current code using the library should continue to work OK. Because of all the code changes, there may be teething troubles. I'll try to watch the mailing lists and regression tests closely for the next couple of weeks, and respond to problems ASAP. --Beman

1 0

asio queue OOM stress test
by christopher baus 16 Dec '05

16 Dec '05

I wrote a little program to stress the queue under low memory situations, and to demonstrate that there is no upper bound on memory usage when using async function calls. The first time I ran it under Linux the kernel killed mysqld and pretty much hung the system. The second time the kernel killed the test program. In neither run could I recover from the OOM error, which is one reason I wrote this. I want to demonstrate that under Linux you often can't catch exceptions from new, because your program is dead before the exception is thrown. Some might consider that a limitation of over committing memory managers like Linux and FreeBSD's, but that's how it works, and I think we are stuck with it. What is really interesting is on Windows, as I expected, I did get the exception and started running the handlers, but the program deadlocked each time I ran it after running about 600k function calls. I'm starting to wonder if this a bug unrelated to the OOM situation. #include <iostream> #include <boost/bind.hpp> #include <boost/asio.hpp> void doit(unsigned long* count) { ++(*count); if(!(*count % 100000)){ std::cout<<*count<<std::endl; } } int main(int argc, char* argv[]) { boost::asio::demuxer d; unsigned long call_count = 0; unsigned long post_count = 0; // // post messages until we run out memory and then // run them. try{ std::cout<<"posting events..."<<std::endl; for(post_count = 1;;++post_count){ // // Internally allocates queueing structures, and // a copy of the functor returned by bind. d.post(boost::bind(doit, &call_count)); if(!(post_count % 100000)){ // // print out something to show that // we are still alive. std::cout<<post_count<<std::endl; } } } catch(...){ // // Let's not do anything here in case we throw again. } // // There is a reasonable chance we might throw here, // because we are pretty much out of memory. Might // want to comment this line out. std::cout<<"caught exception. post_count: " <<post_count<<std::endl; try{ d.run(); } catch(...){ std::cout<<"caught exception running events."<<std::endl; // // Can't really do anything graceful here like continue // to handle connected sockets. // // I could try to call run again, but I doubt that's safe. // as we could have been mucking with internal structures // when the exception let go. } std::cout<<"all done"<<std::endl; std::cout<<"post_count: "<<post_count <<std::endl; std::cout<<"call_count: " <<call_count<<std::endl; return 0; }

3 10

Re: [boost] [serialization][bug]
by Oleg Abrosimov 16 Dec '05

16 Dec '05

It is really strange... I'm using boost 1.33.1 the compiler is MS VC 7.1.3088 In the file attached you can find VC solution and project (with modified demo file) that doesn't compile for me. Please, try to compile it. And if anyone else can do it under VC 7.1, please try it too. Thank you in advance, Oleg Abrosimov //------------------------------------- This was an issue with 1_33_0 that I believe was resolved with 1_33_1. I just tested this and it compiles with my VC 7.1 compiler with no errors. Looking at your code it seems that you're using 1_33_1 so I can't explain why we might be getting different results. Robert Ramey

2 1

[test] self-registration facilities
by Gennadiy Rozental 16 Dec '05

16 Dec '05

Hi, This message should serve as a temporary source of information regarding self registration faculties of Boost.Test. I will include the complete docs in next update. Here is the list of all available macros: ---------------------------------------------------- BOOST_AUTO_TEST_CASE( test_name ) ---------------------------------------------------- Simple free function like test case. Use like this: BOOST_AUTO_TEST_CASE( test ) { // test body } ---------------------------------------------------- BOOST_FIXTURE_TEST_CASE( test_name, F ) ---------------------------------------------------- Simple free function like test case. Employs struct F as a fixture. Use like this: struct F { F() { /* setup here */} ~F() { /* teardown here */} int i; }; BOOST_FIXTURE_TEST_CASE( test, F ) { // test body // i could be referenced here } ---------------------------------------------------- BOOST_AUTO_TEST_CASE_EXPECTED_FAILURES( test_name, n ) ---------------------------------------------------- Use this to specify number of expected failures in test case. Use like this BOOST_AUTO_TEST_CASE_EXPECTED_FAILURES( test1, 1 ) BOOST_AUTO_TEST_CASE( test1 ) { BOOST_CHECK( 2 == 1 ); } ---------------------------------------------------- BOOST_AUTO_TEST_CASE_TEMPLATE( test_name, type_name, type_list ) ---------------------------------------------------- Free function test case template for generating a test case for each type in type list. Use like this: typedef boost::mpl::list<char,int,float,double> test_types; BOOST_AUTO_TEST_CASE_TEMPLATE( test, T, test_types ) { // Body here; T is type under test } ---------------------------------------------------- BOOST_AUTO_TEST_SUITE( suite_name ) BOOST_AUTO_TEST_SUITE_END() ---------------------------------------------------- By default all test cases are registered in master test suite. There macros are used for constructing hierarchical test tree like this: BOOST_AUTO_TEST_SUITE( suite1 ); // this test case belongs to suite1 test suite BOOST_AUTO_TEST_CASE( test1 ) { // body here } BOOST_AUTO_TEST_SUITE_END(); ---------------------------------------------------- BOOST_FIXTURE_TEST_SUITE( suite_name, F ) ---------------------------------------------------- Similar to the BOOST_AUTO_TEST_SUITE, but force all the test cases within this test suite to use struct F as a fixture. BOOST_FIXTURE_TEST_SUITE( s, F ) // this test case use F as a fixture BOOST_AUTO_TEST_CASE( test1 ) { // body here } BOOST_AUTO_TEST_SUITE_END() ---------------------------------------------------- BOOST_GLOBAL_FIXTURE( F ) ---------------------------------------------------- Facilitate an ability to do global setup/teardown. Setup is invoked before any testing is started. Teardown - when all is done. Use like this: struct MyConfig { MyConfig() { std::cout << "global setup\n"; } ~MyConfig() { std::cout << "global teardown\n"; } }; BOOST_GLOBAL_FIXTURE( MyConfig ) You may have as many global fixtures as you want. To automatically setup init function define BOOST_TEST_MAIN in you main test module (in only one module in multi module test program) Alternatively you could define it explicitly. For example you could use it to rename a master test suite. Regards, Gennadiy

2 6

boost infrastructure questions
by Stefan Seefeld 16 Dec '05

16 Dec '05

For quite a while people have been discussing various enhancements to the boost infrastructure. What are these projects up to ? Notably: * Are there still plans to move to subversion ? If so, what is missing ? * What is the status of the move to boost.build version 2 ? * Are there still plans to 'officially' use buildbot to coordinate builds, test runs, and checkins ? I find it quite hard to interpret the regression test matrix since it is't obvious which test runs are in really in sync with the latest revision (or any revision, for that matter), and because there doesn't appear to be an obvious way to distinguish real test failures from failed prerequisites (incl. configuration errors, unsufficient resources, etc.) Thanks, Stefan

6 7

Re: [boost] asio formal review begins
by christopher baus 16 Dec '05

16 Dec '05

> What I did was to have the main thread accept incomming requests, and put > them in the queue, whereas a number of worker threads was taking these > requests from the queue, and execute them. I don't see how this model can > be absolutely inferior in all possible contexts. > I think your argument regarding the sync interface *might* be valid for other reasons, but I want to point out that this architecture only scales if you have an async front end that reads and writes from the socket and parses the connection into tasks in the main thread, and then handles these tasks synchronously in a thread pool. Say for instance a web server which handles HTTP request pipelines asynchronously and database connections synchronously. If the system contains long lived connections, then you are basically only servicing the number of connections in your thread pool, and there is a good chance the other connections will timeout. I do agree with Peter. If asio leads developers away from thread/connection architectures, then it has succeeded. As with a lot of technologies that are shiny and new, I think the industry has put too much emphasis on threads as the solution to all concurrency problems. I think we understand the problems better now. Christopher

4 4

asio review: are we still on topic?
by Arkadiy Vertleyb 16 Dec '05

16 Dec '05

Guys, This general discussion about which approach to networking is superior is very interesting (and useful for me personally), but I think we moved off topic. I think Chris (the author of asio) agreed that both approaches are valid, and claimed he gives both of them appropriate attention. So now, since I pointed to what I believe is a problem with the synchronous interface in asio (socket dependency upon the demuxer), doesn't the critique of synchronous approach sounds a little bit like: "It is inferior anyway, so it's a goot thing that it's not clean" ? Because nobody yet tried to convince me that socket dependency upon demuxer (even if for synchronous IO only) is a good thing. Regards, Arkadiy

5 4

[filesystem] Additional Decomposition Request
by me22 16 Dec '05

16 Dec '05

Hi all, The discussion a little bit ago regarding using path's grammar in a custom virtual filesystem reminded me of my experience when I made a simple unpacking program using boost::filesystem::path. One thing that wasn't as trivial as I might have liked was descending recursively down the internal hierarchy. It would have been nice to have head() and tail() decomposition functions ( though with possibly better names ) analogous to the branch_path() and leaf() functions, but for the front of the path. That way I could have simply looked for the head() element at the current level and recursed with tail(). ( Though I ended up doing it by having my get_file function delegate to one that worked with an iterator range, which is more efficient, but... ) The implementation is, of course, fairly trivial, something along the lines of the following: std::string path::head() { if ( empty() ) return std::string(); return *begin(); } path path::tail() { if ( empty() ) return path(); return std::accumulate( next(begin()), end(), std::divides<path>() ); } And the obvious has_head() and has_tail(). The usage would be something similar to: file &directory::get_file(boost::filesystem::path p) { if ( !p.has_tail() ) return files[ p.head() ]; return subdirectories[ p.head() ].get_file( p.tail() ); } Assuming appropriate ( not part of boost ) directory and file classes. Opinions? Interest? - Scott McMurray P.S. I'm particularly fond of the accumulate/divides hack :P

1 0