March 2010 - Boost - lists.boost.org

[log] Boost.Log formal review is over
by Vladimir Prus 18 Mar '10

18 Mar '10

Boosters, The forward review of the Boost.Log library is now officially over. Big thanks to Andrey for a solid proposal, and for everybody who participated in review. I plan to announce review results in a week from now, on Mar 25. Thanks, Volodya

1 0

[unordered] unordered_set::erase() complexity bug?
by Stefan Strasser 18 Mar '10

18 Mar '10

I think there is something very wrong with boost's implementation of unordered_set/map. 1 unordered_set<int> s; 2 for(int c=0;c<nr;++c) s.insert(c); 3 for(int c=nr-1;c>=0;--c) s.erase(s.find(c)); results: nr: time 20000: 0m0.244s 40000: 0m0.940s 60000: 0m5.584s 80000: 0m4.232s 100000: 0m34.814s 120000: 0m33.486s the function that's causing this is erase(), which is specified to be O(1) average, O(n) worst. I can't think of a reason why it should be O(n) in this case (perfect hash, erase doesn't rehash). but even if it was, the results should be at worst quadratic. load_factor() is below 1, bucket_count(x) is 1. if line 3 is changed to: 3 for(int c=0;c<nr;++c) s.erase(s.find(c)); 20000: 0m0.008s 40000: 0m0.016s 60000: 0m0.028s 80000: 0m0.032s 100000: 0m0.040s 120000: 0m0.044s as you would expect. even if I do the very same thing as above using Boost.Intrusive (using a load factor of 1), I get expected results: struct element : intrusive::unordered_set_base_hook<>{ ... for(int c=0;c<nr;++c) s.insert(*new element(c)); for(int c=nr-1;c>=0;--c) s.erase(s.find(element(c))); 20000 0m0.008s 40000 0m0.016s 60000 0m0.024s 80000 0m0.024s 100000 0m0.032s 120000 0m0.040s but surprisingly enough, the TR1 implementation shipped with libstdc++ also has a problem like that: 20000: 0m0.368s 40000: 0m1.448s 60000: 0m1.668s 80000: 0m7.160s 100000: 0m7.808s 120000: 0m13.349s am I missing something or is there really the same bug in 2 major implementations of unordered_set? it effectively caused a deadlock in some of my code. I'm using 1.38. no bug report in Trac.

20 48

[date_time] memory leak?
by Bartosz Marcinkiewicz 18 Mar '10

18 Mar '10

Hello, Code: #include <iostream> #include <boost/date_time/posix_time/posix_time.hpp> int main() { std::cout << boost::posix_time::microsec_clock::local_time() << std::endl; return 0; } Running through valgrind yields this: ==5963== ==5963== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 7 from 2) ==5963== malloc/free: in use at exit: 1,595 bytes in 27 blocks. ==5963== malloc/free: 63 allocs, 36 frees, 7,878 bytes allocated. ==5963== For counts of detected errors, rerun with: -v ==5963== searching for pointers to 27 not-freed blocks. ==5963== checked 173,784 bytes. ==5963== ==5963== 619 bytes in 20 blocks are possibly lost in loss record 7 of 7 ==5963== at 0x4A06FFC: operator new(unsigned long) (vg_replace_malloc.c:230) ==5963== by 0x352C0A0140: std::string::_Rep::_S_create(unsigned long, unsigned long, std::allocator<char> const&) (in /usr/lib64/libstdc++.so.6.0.12) ==5963== by 0x352C0A0D64: (within /usr/lib64/libstdc++.so.6.0.12) ==5963== by 0x352C0A0F01: std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(char const*, std::allocator<char> const&) (in /usr/lib64/libstdc++.so.6.0.12) ==5963== by 0x408AD7: boost::date_time::date_generator_formatter<boost::gregorian::date, char, std::ostreambuf_iterator<char, std::char_traits<char> > >::date_generator_formatter() (in /home/bma/c++/boost_dt) ==5963== by 0x407B20: boost::date_time::time_facet<boost::posix_time::ptime, char, std::ostreambuf_iterator<char, std::char_traits<char> > >::time_facet(unsigned long) (in /home/bma/c++/boost_dt) ==5963== by 0x40494B: std::basic_ostream<char, std::char_traits<char> >& boost::posix_time::operator<< <char, std::char_traits<char> >(std::basic_ostream<char, std::char_traits<char> >&, boost::posix_time::ptime const&) (in /home/bma/c++/boost_dt) ==5963== by 0x402AA0: main (in /home/bma/c++/boost_dt) ==5963== ==5963== LEAK SUMMARY: ==5963== definitely lost: 0 bytes in 0 blocks. ==5963== possibly lost: 619 bytes in 20 blocks. ==5963== still reachable: 976 bytes in 7 blocks. ==5963== suppressed: 0 bytes in 0 blocks. ==5963== Reachable blocks (those to which a pointer was found) are not shown. ==5963== To see them, rerun with: --leak-check=full --show-reachable=yes This is boost-1.37.0-9.fc11 (fedora core 11), gcc 4.4.1, x86-64 platform. Is it memory leak? Best regards, bma. -- Bartosz Marcinkiewicz tel. (+48) 516 034 243 e-mail: bma(a)epcon.pl, bartosz.marcinkiewicz(a)epcon.pl EPCON Sp. z o.o. www.epcon.pl adres: 53-407 Wrocław, ul. Gajowicka 114/116 tel. (+48) 71 361 75 55 ext.36 fax (+48) 71 361 74 44 NIP PL 899-24-92-498 KRS 0000190187 REGON 932963731 Kapitał zakładowy: 500 000 PLN

3 3

Re: [boost] [unordered] unordered_set::erase() complexity bug?
by Christopher Crickmar 18 Mar '10

18 Mar '10

Given that the next iterator is meaningless for an unordered container and that cached_begin_bucket_ is always valid would it be bad manners to always return begin() ? Chris Crickmar

2 2

[multi_index] hashed_index erase(iterator) performance issue
by Brad Higgins 17 Mar '10

17 Mar '10

Hello, I have encountered a performance issue with multi_index's hashed_index erase(iterator) routine. The erase(iterator) call returns an iterator to the next element in the data structure. When using this routine in a sparsely populated, large sized hash, it takes a very long time to complete. For example, in my program, I have rehashed the hash table to 5 million buckets, and added 600,000 elements. Adding the elements takes a matter of seconds. Then, I delete the elements. Deleting the elements takes hours. I have profiled the code, and it looks to me like the performance hit is rooted in the increment of the iterator by the erase(iterator) call. Since the hash is sparsely populated, the increment must advance over many empty buckets for each erase(iterator) call, in order to advance to the next valid element. My question is, why does the erase(iterator) function return an iterator at all? (In comparison, sgi's stl::hash_map's erase() returns void.) The hash is unordered, so the returned iterator is a random value, which I think decreases its usefulness (I expect most clients will discard the returned iterator.) Additionally, the same behavior can be achieved by clients who desire this behavior if they post-increment their iterator argument themselves. That is: // After this line, cur_itr's initial element will be erased, // AND cur_itr will be updated to the next element in the hash table hash_table.erase(cur_itr++); Is there some way to remove elements from the hash without taking this iterator increment hit? Would it be possible to add an erase routine (of some other name) which accepts an iterator argument, but does not automatically increment the argument? Thanks, Brad

3 3

[log] Comments
by Vladimir Prus 17 Mar '10

17 Mar '10

Hello, I have a few comments and questions about the Boost.Log library currently being reviewed. This email is not actually a review as such. It will help if I explain where I'm coming from. I am most familiar with two logging mechanism. One is from KDE libraries. There, you have severities and component ids -- which are just integers. So, source code might have kDebug(9012) << "doing something"; There's GUI to configure which components have logging enabled, and there's a way to omit explict component id. Another logging mechanism is from Eclipse. There, you specify severity and component id -- this time a string. In KDE, output goes to a console. In Eclipse -- to a log file. I would those mechanism to be sufficient for all my needs. Looking at the proposed libraries, I see two major issues: * There's no default setup that is comparable with the two mechanisms above. BOOST_LOG_TRIVIAL has only severity and it appears adding component information is complicated. In a sense, it looks like the library provides building blocks, but not a solution I can immediately use. * I think the library elegantly uses various facilities like Boost.Parameter and lambda expressions to provide an easy interface. Unfortunately, it seems like it's failing to provide any other interface for folks who might be unhappy with lambdas and named parameters. There's a more detailed explanation of those major issues: * It appears that out of the box, the proposed library does not address this component/severity style of logging that I've explained above. The proposed solutions at: http://permalink.gmane.org/gmane.comp.lib.boost.devel/200793 sound relatively complex. * The example of specifying filter without lambda functions given in http://permalink.gmane.org/gmane.comp.lib.boost.devel/200793 is frankly scary. I believe that in most languages and environments I worked with, lambda functions are just convenient alternative to defining the same function explicitly. Why is in Boost.Log, lambda expressions are blessed with extra convenience? * Why can't I use the library without named parameters? Surely, there are low-tech ways to specify things. * Why does channel_logger has a *single* parameter, and still requires that you pass it via named parameter. That is: src::channel_logger<> cl("foobar"); produces incomprehensible error message. Other concerns: * The library went too far with namespaces. Why are *six* nested namespaces necessary? This appears to be just asking for extra typing for no good reason. * Why is trivial logging printing the sequental number of each log message? I don't think this helps anything. Also, it prints the thread "number", on my system a hex string. I am not sure thread id should be part of trivial logging. Finally, I think by default, everybody things of logging as glorified std::cout, so trivial logging better print messages to console, not to a file. * Why does the library have light_rw_mutex? I would really prefer if everything thread-related be located inside Boost.Thread, so that at least the code can be readily audited. * It appears that the default behaviour of fmt::attr is to throw is the attribute with such name is not found. This seems like a bad choice to me. A logging library should never fail -- it is the last resort at diagnosing a problem in the field, and if it throws by default, it's useless. * I have used the attached script to measure code size impact. The script basically creates a test file with a single function calling BOOST_LOG_TRIVIAL and measures the size of that function depending on the number of calls. I've got 139 bytes per call for 'g++ -Os' and 174 per call for 'g++ -O3', which seems nice. This is function code only. When counting file size I get 324 and 514 respectively. For reference, kDebug gives 136 and 328 of file_size_bytes per call. I guess these results are OK. * I see lots of warnings in slim_string.cpp, e.g: ../../../../libs/log/src/slim_string.cpp:237: warning: control reaches end of non-void function I'd recommend working with maintainer of boost/throw_exception.hpp to add noreturn attributes on gcc, and silence the warning in other ways on other compilers. * Building the library with threading=single results in unresolved references to pthread functions. I belive that library should not use any sync mechanisms in this case. Thanks, Volodya

10 70

[log] Formal review
by Alexander Arhipenko 17 Mar '10

17 Mar '10

Boost.Log library formal review. > Please explicitly state in your review whether the library should be accepted. Yes, library should be accepted. Some explanation here. What means 'boost library' to me? It's very reliable, portable, efficient, sexy (in sense of code style) component that could satisfy from 90 to 95 % of regular user cases. Even if someone has an issue with such component - this issue could be resolved quickly (up to 1 hour). This quickness usually explained by clean library design and also by rich community support (the last sentence I've made based on my own experience with boost). IMHO, Boost.Log nicely fits into all the written above. Andrey spent many months (I've put an eye on Boost.Log in August 2008) polishing library's design, implementation, improving documentation. It would be nice to see the result of such huge efforts in Boost distribution. - What is your evaluation of the design? The overall design seems to be very orthogonal and does not mix different things together. - What is your evaluation of the implementation? I didn't dig deeply (except some functionality related to log rotation). At first glance, it's high quality, modern c++ code. It doesn't take too much time to understand it and make modifications. - What is your evaluation of the documentation? The first thing I except from any documentation - it's a "quick jumps" scenarios, i.e. workable small code snippets that compiles, works and give the user brief explanation on how to perform usual tasks using the library. I've found it in the latest docs and that was nice. Tutorial section is organized in 'from simple to complex things' form. This is very nice. Overall quality of the documentation is fine to me. - What is your evaluation of the potential usefulness of the library? Every application could really benefit from using logging. Especially server ones, that are running 24/7 without frequent human interaction. But I'm saying banal things now. In short, potential usefulness of such library is very high. - Did you try to use the library? With what compiler? Did you have any problems? Yes, I've tried it. gcc 4.1.2 (RHEL 5 x86_64, RHEL 5 i686). msvc 9 (Windows i686). The only problem I had it was warnings on gcc. But that was RC3 of the library, I guess Andrey addressed those issues in the latest version. - How much effort did you put into your evaluation? A glance? A quick reading? In-depth study? We've integrated Boost.Log RC3 into our server application (in the end of January 2008). Now it's running in several production environments. I've spent several days writing tiny wrapper for Boost.Log. This wrapper was required due to reasons above: 1) I was not sure about the future: whether we would like to switch to the different logging library etc., nowadays answer to this question is categorical NO 2) All the components of application should have common logging formatting, logging levels, also some 'shortcuts' were needed (such as namespace aliases etc) Also, we did some Boost.Log profiling with valgrind (memcheck tool). As far as I remember, none issues were found. - Are you knowledgeable about the problem domain? Knowledgeable from the end-user point of view. I know, what I want to have from logging (e.g., various severities, filtering, log files rotation, thread-safety). - Questions I have one question in regards to log rotation. (I guess, I was asking it last year when RC3 was presented). Is it possible to setup log rotation in a following way: file.log - earlies log file (today) file1.log - older log file (yesterday) file2.log - oldest log file (the day before yesterday). This way Apache, Samba etc. performs rotation. Also, this rotation behavior is default in Python logging module. Also, this behavior is default in my wolrd ;). Many thanks to Andrey for Boost.Log contribution. Many thanks to Volodya for being a review manager. Regards

2 1

[log] How to set/reset a channel-specific filter
by Andreas Huber 17 Mar '10

17 Mar '10

Hi We're currently reviewing Boost.Log and are wondering what is the best way to implement channel-specific filters. More precisiely, we are currently using the following "objects": - 2 sinks (one to debug output window, one to file) - several severity_channel_logger_mt objects, all declared with BOOST_LOG_DECLARE_GLOBAL_LOGGER, each of these is guaranteed to have a unique channel name For each of the channels, we would like to set a filter that only applies to records logged through the given channel. Since we can't seem to set a filter on the logger itself, we're wondering what is the best way to accomplish this? It seems as though both the core as well as the sinks only accept one filter. So, it seems we would have to a) assemble a filter expression that includes all the channels we'd like to filter on. *Or*, we could b) add exactly two sinks per logger object and set the desired filter on both sinks. Of course, we'd still want to only log to one file, so the question arises whether it is safe to have multiple sinks write to one file. If we go with method a), we'd need a way to enumerate all global loggers. If we go with b), we'd need a method to enumerate all the sinks in the system. We have not found a way to accomplish any of these tasks, how do we do this? Thanks & Regards, -- Andreas Huber When replying by private email, please remove the words spam and trap from the address shown in the header.

2 7

[python]missing file
by K. Noel Belcourt 17 Mar '10

17 Mar '10

Hi, Trunk tests are dying with this error. error: Unable to find file or target named error: 'builtin_converters_ext' error: referred from project at error: '../libs/python/test' -- Noel

1 0

[exception] Unreachable returns (was [log] Comments)
by Daniel James 17 Mar '10

17 Mar '10

On 15 March 2010 10:45, Vladimir Prus <ghost(a)cs.msu.su> wrote: > > * I see lots of warnings in slim_string.cpp, e.g: > > ../../../../libs/log/src/slim_string.cpp:237: warning: control reaches end of non-void function > > I'd recommend working with maintainer of boost/throw_exception.hpp to add noreturn attributes on > gcc, and silence the warning in other ways on other compilers. I think Boost.Exception already does that (http://www.boost.org/boost/exception/detail/attribute_noreturn.hpp) but this is something I've been meaning to ask about, as I had to deal with it for iostreams. Our normal solution is to use BOOST_UNREACHABLE_RETURN (http://tinyurl.com/yjjdzsh) but that doesn't work for compilers that don't understand a 'noreturn' attribute. I hacked together my own version (http://www.boost.org/boost/iostreams/detail/config/unreachable_return.hpp) but it's only a short term solution and could break if throw_exception changes. It would be nice to have a proper solution in Boost.Exception, or maybe there already is one and I missed it? Daniel

2 2