August 2012 - Boost - lists.boost.org

[thread] Memory usage of threads on 64bit linux
by Conrad Mercer 29 Aug '12

29 Aug '12

Hey, I have been using boost threads on 32bit linux for some time and am very happy with their performance so far. Recently the project was moved to a 64bit platform and we saw a huge increase in memory usage (from about 2.5gb to 16-17gb). I have done profiling and found that the boost threads are the source of the huge allocation. Each thread is allocating about 10x what it was doing on 32bit. I profiled using valgrind's massif and have confirmed the issue using only boost threads in a separate test application. I also tried using std::threads instead and these do not exhibit the large memory allocation issue. I am wondering if anyone else has seen this behaviour and knows what the problem is? Thanks --

4 4

[system] Viral C++11 change would break dependent libraries:-(
by Beman Dawes 28 Aug '12

28 Aug '12

Ticket #7278 asks that Boost.System add C++11 noexcept support. A reasonable request, and I implemented it this morning by adding BOOST_NOEXCEPT at the appropriate places specified in the standard. But there is a problem. noexcept is viral, so any classes that inherit from a class that uses noexcept must be changed to also use no except. An example: Boost.Filesystem has this class: class codecvt_error_cat : public boost::system::error_category { public: codecvt_error_cat(){} const char* name() const {return "codecvt";} std::string message(int ev) const; }; Compiling with GCC 4.7 -std=c++0x results in this error: ..\..\../boost/system/error_code.hpp:187:32: error: overriding 'virtual const char* boost::system::error_category::name() const noexcept (true)' To fix it, the filesystem class has to be changed: class codecvt_error_cat : public boost::system::error_category { public: codecvt_error_cat(){} const char* name() const BOOST_NOEXCEPT {return "codecvt";} std::string message(int ev) const; }; I've held off committing the Boost.System changes to trunk until we've had a chance to discuss how to approach this potential break to an unknown number of Boost and user libraries. Ideas? --Beman

5 7

[contract] Broken postconditions on throw
by Andrzej Krzemienski 28 Aug '12

28 Aug '12

Hi Lorenzo, I am now now playing with Contract library and trying to break it. I checked that the framework checks the class invariant even when member function exits via an exception. This is an expected behavior and the framework is working fine. However, I note that the precondition of a (non-member) function is not checked when the function exits via throwing an exception. See the following program: # include <contract.hpp> double get_random_weight() { return -1.0; // erroneous implementation } CONTRACT_FUNCTION ( void (set_weight)( (double&) w ) postcondition(w >= 0.0) ) { w = get_random_weight(); if (w < 0.0) throw 1; // (*) } int main() try { double w = 0.0; set_weight(w); } catch (...) { std::cerr << "CAUGHT EXCEPTION" << std::endl; } Function set_weight() is intended to assign its (output) argument with a value representing weight (in the physical sense). We guarantee that the value we set is always non-negative. The function clumsily attempts to verify the condition itself (the line with *). It signals the failure with an exception, but unfortunately fails to satisfy the postcondition. I expect that this situation should be reported as a postcondition violation. But no violation is reported, and the program successfully outputs "CAUGHT EXCEPTION". Regards, &rzej

4 11

[1.52.0] Open for merging
by Marshall Clow 28 Aug '12

28 Aug '12

The release branch (https://svn.boost.org/svn/boost/branches/release) is open for merging all stable changes from trunk, including bug fixes and major upgrades to existing libraries. Breaking changes should be coordinated via the mailing list with the developers of the other libraries affected. New libraries may be added, but require permission of a release manager. In other words, merging *does not* require permission from a release manager, except for new libraries. Boost Calendar with the release schedule: http://www.boost.org/development/index.html Release schedule explanation: http://svn.boost.org/trac/boost/wiki/ReleaseSchedule Release practices for developers: http://svn.boost.org/trac/boost/wiki/ImprovingPractices -- The Release Managers, Beman Dawes Daniel James Eric Niebler Marshall Clow Rene Rivera Vladimir Prus

3 2

[contract] a possible future extension
by Andrzej Krzemienski 28 Aug '12

28 Aug '12

Hi Lorenzo, This is definitely outside the scope of the review, but I was wondering whether it would be feasible for the macros in Boost.Contract to handle base class aliases, as described in this proposal: http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2009/n2881.html The positive "side effect" of your framework is that it has the potential to add various missing language features. I am wondering if these aliases proposed in N2881 are worth the effort. Regards, &rzej

2 1

a (useful?) crtp trick
by pavel 28 Aug '12

28 Aug '12

hi all i post this in the hope someone finds this technique useful to make long story short, here is a snippet: template<typename t> //tag deducing util struct base_tag { typedef decltype(get_tag((t*)42)) type; }; template<typename type, typename tag = typename base_tag<type>::type> struct Base //crtp base class { friend tag get_tag(type*); //helper declaration, never defined }; template<typename type> //no mention of tag void f(const Base<type>&) //tag is deduced automatically {} struct Foo : Base<Foo, void> //derived type {}; int main() { Foo foo; f(foo); } the idea is that if we have an associated type (or types) along with the "real" type in the crtp pattern we can make that associated type deduced automatically and even pretend there is no such tag leaving it e.g. for advanced usage more on this here: http://programmizm.sourceforge.net/blog/2012/because-we-can -- Pavel P.S. if you notice a grammar mistake or weird phrasing in my message please point it out

3 7

release manager for 1.52?
by Oliver Kowalke 28 Aug '12

28 Aug '12

Who is the release manager for 1.52?

2 1

[atomic] lockfree failure on msvc 64-bit
by Christoph Macheiner 27 Aug '12

27 Aug '12

As indicated by http://www.boost.org/development/tests/trunk/developer/output/cm_x64-boost-…. In boost/atomic/detail/interlocked.hpp, BOOST_ATOMIC_LLONG_LOCK_FREE must be defined to 2 if _WIN64 is defined (cas64strong.hpp is included afterwards). Because the library has not yet been released, I didn't bother to create a ticket and because the change is so trivial I didn't create a patch as well (if either or both is required or wanted, please let me know). Question: If I recall correctly, long long can be made lockfree also in 32-bit. If I am right, is there a reason why it hasn't been done? Thanks. --Christoph.

1 0

memory corruption in async_read_until using asio::ssl
by Jim Page 27 Aug '12

27 Aug '12

Afternoon All I am seeing an occasional memory corruption bug, using valgrind, in our web server application that is using boost::asio::ssl, and it seems to be related to calls to async_read_until, where several chunks of data are required before the terminating regex is found. What appears to be happening is that engine::perform() causes a write (via ssl3_read_bytes()) on a deleted memory region, the buffer into which decrypted data is written. This region was delete'd via invocation of the detail::io_op::handler, and in turn via read_until_delim_op::operator(), due to prepare() being called on the streambuf, and a std::vector<char> is delete'd in order to create a larger one. It seems like one of the chain of handlers is using a pointer which is becomes invalid when the streambuff is expanded after failing to find the regex terminator after one or more reads, and prepare() is called before the next async_read_some() on the underlying stream. I have included the valgrind output at the end - sorry for the large size. I am pretty certain that I have ruled out the obvious (multiple simultaneous attempts to read using the same streambuf), and I am only using one thread for the io_service. This seems a pretty common potential failure mode, so perhaps I am barking up the wrong tree - I'd appreciate anyone's opinion before I start a potentially tricky debug effort requiring modification to the boost libraries on our production servers. I'm using boost 1.49. There's a lot of optimization on the application which is why some calls are optimized out of the valgrind dump. The machines are running Centos 5.8, x86_64. Compiler is gcc 4.4.6. Thanks for looking Jim Page ==29870== Thread 22: ==29870== Invalid write of size 1 ==29870== at 0x4A088F8: memcpy (mc_replace_strmem.c:587) ==29870== by 0x3400821EDE: ssl3_read_bytes (in /lib64/libssl.so.0.9.8e) ==29870== by 0x340081E750: ??? (in /lib64/libssl.so.0.9.8e) ==29870== by 0x66A75B: boost::asio::ssl::detail::engine::perform(int (boost::asio::ssl::detail::engine::*)(void*, unsigned long), void*, unsigned long, boost::system::error_code&, unsigned long*) (engine.ipp:219) ==29870== by 0x7E3326: boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio ::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::ssl::detail::read_op<boost::asio::mutable_buffers_1>, boost::asio::detail::rea d_until_expr_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<bo ost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> > >, std::allocator<char>, boost::basic_regex<char, boost::regex_traits<char, boost::cpp_regex_traits<char> > >, boost::_bi::bind_t<void, boost::_mfi::mf2<void, HttpConnection, boost::system::error_code const&, char const*>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<HttpConnection> >, boost::arg<1> (*)(), boost::_bi::value<char const*> > > > >::operator()(boost::system::error_code, unsigned long, int) (engine.ipp:151) ==29870== by 0x7E45C2: boost::asio::detail::wait_handler<boost::asio::ssl::detail::io_op<boost::asi o::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::ssl::detail::read_op<boost::asio::mutable_buffers_1>, boost::asio::detail::read_until_expr_op<boost::asio::ssl::stream<boost::asio ::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> > >, std::allocator<char>, boost::basic_regex<char, boost::regex_traits<char, boost::cpp_regex_traits<char> > >, boost::_bi::bind_t<void, boost::_mfi::mf2<void, HttpConnection, boost::system::error_code const&, char const*>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<HttpConnection> >, boost::arg<1> (*)(), boost::_bi::value<char const*> > > > > >::do_complete(boost::asio::detail::task_io_service*, boost::asio::detail::task_io_service_operation*, boost::system::error_code const&, unsigned long) (bind_handler.hpp:46) ==29870== by 0x619783: boost::asio::detail::task_io_service::run(boost::system::error_code&) (task_io_service_operation.hpp:37) ==29870== by 0x79F4FE: BayeuxServer::run() (io_service.ipp:59) ==29870== by 0x364800DB8E: ??? (in /usr/lib64/libboost_thread.so.1.49.0) ==29870== by 0x3AE300677C: start_thread (in /lib64/libpthread-2.5.so) ==29870== by 0x3AE20D325C: clone (in /lib64/libc-2.5.so) ==29870== Address 0x1b508610 is 0 bytes inside a block of size 1,024 free'd ==29870== at 0x4A05A33: operator delete(void*) (vg_replace_malloc.c:346) ==29870== by 0x658E54: std::vector<char, std::allocator<char> >::_M_fill_insert(__gnu_cxx::__normal_iterator<char*, std::vector<char, std::allocator<char> > >, unsigned long, char const&) (new_allocator.h:95) ==29870== by 0x68B805: boost::asio::basic_streambuf<std::allocator<char> >::reserve(unsigned long) (stl_vector.h:851) ==29870== by 0x7E2EF2: boost::asio::detail::read_until_expr_op<boost::asio::ssl::stream<boost::asio ::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> > >, std::allocator<char>, boost::basic_regex<char, boost::regex_traits<char, boost::cpp_regex_traits<char> > >, boost::_bi::bind_t<void, boost::_mfi::mf2<void, HttpConnection, boost::system::error_code const&, char const*>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<HttpConnection> >, boost::arg<1> (*)(), boost::_bi::value<char const*> > > >::operator()(boost::system::error_code const&, unsigned long, int) (basic_streambuf.hpp:207) ==29870== by 0x7E32DE: boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio ::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::ssl::detail::read_op<boost::asio::mutable_buffers_1>, boost::asio::detail::read_until_expr_op<boost::asio::ssl::stream<boost::asio ::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> > >, std::allocator<char>, boost::basic_regex<char, boost::regex_traits<char, boost::cpp_regex_traits<char> > >, boost::_bi::bind_t<void, boost::_mfi::mf2<void, HttpConnection, boost::system::error_code const&, char const*>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<HttpConnection> >, boost::arg<1> (*)(), boost::_bi::value<char const*> > > > >::operator()(boost::system::error_code, unsigned long, int) (read_op.hpp:59) ==29870== by 0x7E493C: boost::asio::detail::reactive_socket_recv_op<boost::asio::mutable_buffers_1, boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio ::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::ssl::detail::read_op<boost::asio::mutable_buffers_1>, boost::asio::detail::read_until_expr_op<boost::asio::ssl::stream<boost::asio ::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> > >, std::allocator<char>, boost::basic_regex<char, boost::regex_traits<char, boost::cpp_regex_traits<char> > >, boost::_bi::bind_t<void, boost::_mfi::mf2<void, HttpConnection, boost::system::error_code const&, char const*>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<HttpConnection> >, boost::arg<1> (*)(), boost::_bi::value<char const*> > > > > >::do_complete(boost::asio::detail::task_io_service*, boost::asio::detail::task_io_service_operation*, boost::system::error_code const&, unsigned long) (bind_handler.hpp:118) ==29870== by 0x613778: boost::asio::detail::epoll_reactor::descriptor_state::do_complete(boost::asi o::detail::task_io_service*, boost::asio::detail::task_io_service_operation*, boost::system::error_code const&, unsigned long) (task_io_service_operation.hpp:37) ==29870== by 0x619783: boost::asio::detail::task_io_service::run(boost::system::error_code&) (task_io_service_operation.hpp:37) ==29870== by 0x79F4FE: BayeuxServer::run() (io_service.ipp:59) ==29870== by 0x364800DB8E: ??? (in /usr/lib64/libboost_thread.so.1.49.0) ==29870== by 0x3AE300677C: start_thread (in /lib64/libpthread-2.5.so) ==29870== by 0x3AE20D325C: clone (in /lib64/libc-2.5.so)

1 0

Boost.Process 0.5 released
by Boris Schaeling 27 Aug '12

27 Aug '12

Boost.Process 0.5 has been released! Documentation and library can be found at: <http://www.highscore.de/boost/process0.5/> Why version 0.5? We have no official process management library in Boost yet. But we had countless drafts since 2006. In order not to get completely lost between all those drafts I gave them version numbers. For example, 0.1 was the very first draft in 2006. 0.31 is the draft documented at <http://www.highscore.de/cpp/process/>. And 0.4 was the draft which was rejected in a formal review in spring 2011. Version 0.5 is the latest attempt to create a process management library. What's new? Version 0.5 has a very different design than all other drafts before. The design was proposed by Jeff Flinn who introduced it briefly at BoostCon 2011 when we gave a presentation on Boost.Process. The goals of this new version are: * A lightweight implementation (no overly complicated code anymore). * A fully extensible library (no need for arbitrary "extension points"). * Support for system-level cross-platform concepts only (no Java-like constructs). Other goals like making the library easy to use were of course also targeted. But these were goals before. The goals above are important as those things have been criticized in earlier drafts and were as far as I see the main reasons why version 0.4 had been rejected. What now? Please have a look at the documentation and think about whether this library could fit into Boost. If you have more time, please download and play around with the new version. Concentrate on the bigger picture: Does the design make sense? Is it easy to understand? Can you do what you want to do? Does it look and feel like modern C++? This is not a formal review. But it would help to get some feedback to see a tendency. After six years, two Google Summer of Code projects, countless drafts and still a lot of interest from many people in a process management library in Boost, we might be able to get somewhere this time. (If you are on Windows: There is a bug in Boost.Iostreams which has been fixed in the release candidate 1 of Boost 1.51.0. See the documentation if you run into a problem when closing pipes.) Anything missing? We had a closed beta test in the previous two weeks with about ten people who had contacted me because of their interest in Boost.Process. I got already some good feedback and have to change a few details. These are only details though and shouldn't stop anyone from using this new version now. If everyone is happy with the overall design and architecture and it's only details we have to discuss, it's already a huge step forward. Last but not least: Boost.Process 0.5 wouldn't exist without the help of Intra2net AG (<http://www.intra2net.com/>). Intra2net AG sponsored a project to support the development of Boost.Process. It was this sponsorship which made it possible to create this new version. Thanks to them! Boris

9 46