- Boost - lists.boost.org

[Beman's endian] explicit native endianes
by vicente.botet 05 Jun '10

05 Jun '10

Hi Beman, I see that there are specific classes for native endian, endian<endianness::native, ...>. I was wondering if instead of defining a separated class we can not define endianness::native depending on the ENDIAN of the host and remove the explicit definition. # ifdef BOOST_BIG_ENDIAN BOOST_SCOPED_ENUM_START(endianness) { big, little, native=big }; BOOST_SCOPED_ENUM_END #else BOOST_SCOPED_ENUM_START(endianness) { big, little, native=little }; BOOST_SCOPED_ENUM_END #endif Will this break the design? Best, _____________________ Vicente Juan Botet Escribá http://viboes.blogspot.com/

4 5

[endian] swap_in_place use case
by vicente.botet 05 Jun '10

05 Jun '10

Hi, I'm looking for a use case where the use of swap_in_place (or whatever is renamed) is not followed/preceded by a copy. When I need to send a message I need to copy the information to send in a buffer. I gues that in this case the use of swap (copy) is the function to use. Can somene help me to identify a real use case on which the swap_in_place can be used to send a message that do not need to be initialized before sending? When I receive a message, I will need to store some of the messages information on application context, store them on local variables or pass them to other functions. All this suppose a copy from the buffer to the different kinds of native storage. I gues that swap (copy) should be used in this case also. The example of Terry reveives a array of ints and makes a swap_in_place, but do nothing with. Someone has proposed to sum all these ints, but in this case, we don't need to swap_in_place and the count. With swap approach, we can count at the same time as we do the conversion, so no need to swap in place. Can somene help me to identify a real use case on which the swap_in_place can be used to reveive a message, without needing to do any additional copy from the swapped buffer to native data? Thanks, _____________________ Vicente Juan Botet Escribá http://viboes.blogspot.com/

3 3

Trunk tests broken
by Belcourt, Kenneth 05 Jun '10

05 Jun '10

Hi, Could I suggest that when the trunk tests are completely broken, as they are now, and the report generation is cycling normally (every 1 to 2 hours) as it is now, that the likely source of breakage is a recent commit to the trunk repository. To reproduce the trunk failure, simply checkout out a current version of the trunk, cd to the status directory, and run bjam. If you do this right now, you'll find this error message. error: Unable to find file or target named error: 'test_diamond_complex.cpp' error: referred from project at error: '../libs/serialization/test' This is what's currently killing trunk testing. Since it's quite easy to reproduce the trunk failure, I think it would help Boost if developers would, when they recognize a problem, either revert the offending commit or ping the library author(s) to let them know about the breakage. Unfortunately I am unable to keep on top of when trunk testing is broken (i.e. it could be several days before I notice it) so if other developers could help diagnose breakage and report it, that would be very helpful. [ Robert, can you please address this issue? ] Thanks. -- Noel

8 17

Re: [boost] [endian] swap_in_place use case
by Dave Handley 04 Jun '10

04 Jun '10

Vicente Botet wrote: > Couldn't the filter be adapted to the endianess of the file and work directly on the disk format? Surely this violates the most important principle of endian swapping - that you should always work in native endian and swap on the boundaries. If I specifically wrote the algorithm to work on the disk endianess I would need to adapt the algorithm to cope with both big and little endian situations; the algorithm would change depending on which platform I compiled on; and every person coding on the algorithm would need to code to be endian aware. This would be a maintenance nightmare. I'm not sure I've been totally clear on what I mean by a filter. I mean a filter in the signal processing sense of the word. So let's set up a problem where we are reading a 10000x10000 image with 4 byte integer gray scale at each pixel and uncompressed (so 400Mb file). This is in big endian format. What I then want to do is convolve a number of kernel with the original image to perform various image transformation. The kernels I'm going to use are, say, the following (Sobel kernels): [ 3 10 3 ] [ 0 0 0 ] [ -3 -10 -3 ] [ 3 0 -3 ] [ 10 0 -10 ] [ 3 0 -3 ] And maybe a few more. Each will be convolved with the image, and since I'm using this for feature detection, all I want to do is create a list of points over a given (absolute) threshold. So I'm creating a resulting data object that is much smaller than my input object. Now here, if I swap on use, then if I run this code on a little endian machine I will end up swapping everything 9 times per 3x3 kernel that I convolve. If I swap up front and then convolve, on a big endian machine I will end up using twice the memory (and copying the whole source file). If on the other hand I swap in place, on both little and big endian machines I have an efficient algorithm. My pseudo-code for this is going to be: Swap in place version: 1) Memory map big file 2) Endian swap in place 3) For each filter: Convolve filter with image, and output results Swap and copy version: 1) Memory map big file 2) Copy entire big file whilst endian swapping 3) For each filter: convolve with copy of image, and output results Swap on use version: 1) Memory map big file 2) For each filter: convolve with disk-endian image, hence for each pixel in image swap pixel and 8 surrounding pixels, multiply given 9 pixels by the filter and output if over threshold. End up swapping each pixel 9 times. Terry Golubiewski wrote: > I disagree. I have demonstrated that... > > 1) endian-on-access can be zero cost in the native-case. > 2) swapping in place is not any safer because the C++ type system cannot > help to determine which portions of an object have been swapped, nor > document the endian properties of a data structure. endian<big, double> > can, and should, be defined to provided portable floating-point transfer and > persistent storage. > I, in turn, also disagree. I'm not quite sure what you did demonstrate with your test case. If by "endian-on-access" you mean copying during the endian swap, then I don't think it is ever zero cost since a copy is always required. It is the same cost as the swap/swap-in-place code if and only if your code requires a copy to take place at some point, at which point both cases require the cost of a copy. To be honest, because your test case didn't separate reading data in from performing the swap, I was really unsure what information it actually conveyed. I would agree with another commenter (Robert I think) who asked for some timing code to be placed around the actual endian swapping to properly time how long the swap took on its own. I don't think the issue with endian swapping code is "safety" since you should really be endian swapping at the boundaries (when the data is read into the machine you should be converting to machine endian) and that will always be "safe". Actually, the big problem I have with the endian types is that you require 2 versions of every structure you are reading in for swapping (one version with endian types and one without); and that you are tempted to defer endian swapping until very late in the code, which means that you are no longer endian swapping at the boundaries. I would really dislike debugging a program that carried around data in the wrong endianness and only swapped it when used. I actually suspect the most common use case of endian types would be as a really short lived temporary storage for the wrong endian data from disk/network/elsewhere, and as such I don't see what you gain with a "type-safe" interface. Dave Handley

1 0

Boost Bug Sprint #3 starts tomorrow!
by Marshall Clow 04 Jun '10

04 Jun '10

The bug sprint will run from tomorrow June 5th until Sunday, June 13th. As of 11:00 AM PST today, there are currently have 992 open tickets. The goal is to classify and fix bugs, and to close Trac tickets. For those just joining in, the tickets live at https://svn.boost.org The instructions (mostly repeated here) for the first bug sprint are at https://svn.boost.org/trac/boost/wiki/BugSprintJune2009 As the bug sprint progresses, I will be posting updates to the list and to https://svn.boost.org/trac/boost/wiki/BugSprintStatusJune2010 What can you do? You can fix bugs • If you are a library maintainer, you can commit your fixes. • If you are not a library maintainer, you can fix a problem on your local machine (and test it!!), and then attach your fix as a patch to the Trac ticket for the library maintainer to commit. If you do this, please provide a test case that exercises the problem; i.e, one that fails w/o your patch, and succeeds afterwards. You can characterize bugs • You can attempt to replicate the bug on your system(s), and attach information that you discover to the Trac ticket. This will make it easier for other people to solve the problem, even if you cannot. • You can write test cases demonstrating the problem, and attach them to the Trac ticket. • You can determine if the bug still exists; sometimes they get fixed, and the ticket never gets closed. In this case, try to identify a checkin that addresses the bug; it's always suspicious when bugs go away "by accident." You can test patches • Some of the tickets in the Trac system contain patches - some are quite old. You can install these patches on your system, and determine if they (a) build, and (b) work. If you do this, please provide detail. Don't just say "works for me" - say "I installed this over revision XXX on system YYY with compiler ZZZ. The patch installed and compiled successfully, and passed all the tests." • You can write test cases for the patches, and attach them to the ticket. You can help implement new features • There are a whole set of "feature requests" in the Trac system. If you have an idea about one of these, you can add comments to the Trac system. You can attempt to implement these new features, and attach the patch to the ticket. (Don't forget the test cases!!). There are several Trac reports that I find helpful for looking at the bugs: Open ticket count by owner: https://svn.boost.org/trac/boost/report/18 Open ticket count by component: https://svn.boost.org/trac/boost/report/19 Open ticket count by type https://svn.boost.org/trac/boost/report/20 (Overwhelmingly, these are 'bugs') Open ticket count by milestone: https://svn.boost.org/trac/boost/report/21 and, of course, the main report at https://svn.boost.org/trac/boost/report/1 Some things I see while looking at these reports: * There are 82 bugs with a milestone of "Boost 1.36.0", which was released 22 months ago. * There are 24 tickets with no owner, of which 15 are 'bugs'. * There are 63 tickets with no milestone attached; we have the "to be determined" milestone for cases where you're not sure when the fix will go in. -- Marshall

1 0

[system] error_code.hpp - questionable includes
by Christian Holmquist 04 Jun '10

04 Jun '10

Hi, I noticed that system/error_code.hpp caused some unexpected increase in compilation time, and looked trough its includes. The following seems not needed: #include <boost/cstdint.hpp> #include <boost/assert.hpp> #include <boost/operators.hpp> #include <boost/noncopyable.hpp> // 1) #include <stdexcept> #include <functional> // 2) 1) noncopyable is used to tell that a class (error_category) with pure virtual functions cannot be copied. This is implicit in the language, so really not needed. Unless there's a compelling reason to make sure derived classes can't be copyable either, but I didn't see any reason for this? I didn't look very close though, could've missed it. 2) inclusion of <functional> seems only needed to have the following: bool operator<( const error_category & rhs ) const { return std::less<const error_category*>()( this, &rhs ); } I can't guess the reason for using std::less instead of simply writing '<', but surely it can be expressed without std::less, no? And before getting the common Booster's answer when asking to remove #includes: Yes, I am aware of precompiled headers. I use them, too. =) Cheers, Christian P.S. Would it be too much to ask for having a separate header for the error_code's ostream operators?

2 2

[test] patch for test_fp_comparisons.cpp
by Jürgen Hunold 04 Jun '10

04 Jun '10

Hi ! According to http://www.boost.org/development/tests/trunk/developer/test.html nearly all compilers fail compiling test_fp_comparisons.cpp. With gcc-4.4 I get: test_fp_comparisons.cpp: In member function ‘void test_close_at_tolerance::test_method()’: test_fp_comparisons.cpp:165: error: reference to ‘fpc’ is ambiguous test_fp_comparisons.cpp:157: error: candidates are: namespace fpc = boost::math::boost::math::fpc; /home/hunold/src/devel/boost/boost/test/floating_point_comparison.hpp:280: error: namespace boost::test_tools::fpc = boost::math::boost::math::fpc; The attached patch removes the offending namespace alias in the test file. Okay to commit ? Yours, Jürgen -- * Dipl.-Math. Jürgen Hunold ! Ingenieurgesellschaft für * voice: ++49 511 262926 57 ! Verkehrs- und Eisenbahnwesen mbH * fax : ++49 511 262926 99 ! Lister Straße 15 * juergen.hunold(a)ivembh.de ! www.ivembh.de * * Geschäftsführer: ! Sitz des Unternehmens: Hannover * Prof. Dr.-Ing. Thomas Siefer ! Amtsgericht Hannover, HRB 56965 * PD Dr.-Ing. Alfons Radtke !

1 0

[test] patch to enable gcc visibility support
by Jürgen Hunold 04 Jun '10

04 Jun '10

Hi ! I've attached a patch which uses the new Boost.Config logic introduced in revision 62140 to enable gcc visibility support. This fixes #2114 for Boost.Test. It is based on the already working patches to Boost.System and testet on gcc-4.4, 4.5 and msvc-9.0 and 10. Okay to commit ? Yours, Jürgen -- * Dipl.-Math. Jürgen Hunold ! Ingenieurgesellschaft für * voice: ++49 511 262926 57 ! Verkehrs- und Eisenbahnwesen mbH * fax : ++49 511 262926 99 ! Lister Straße 15 * juergen.hunold(a)ivembh.de ! www.ivembh.de * * Geschäftsführer: ! Sitz des Unternehmens: Hannover * Prof. Dr.-Ing. Thomas Siefer ! Amtsgericht Hannover, HRB 56965 * PD Dr.-Ing. Alfons Radtke !

2 2

Re: [boost] [endian] swap_in_place use case
by Dave Handley 04 Jun '10

04 Jun '10

Vicente Botet wrote: > I'm looking for a use case where the use of swap_in_place > (or whatever is renamed) is not followed/preceded by a copy. I'm not sure you got an answer to this question. Here is a simple use case. Memory map a network endian file. Swap_in_place. Use. You definitely don't want a copy in this case since your file could easily be very big. Think of the case where your file is huge (say 10Gb), you really don't want to perform a copy and swap since that puts your memory need at 20Gb instead of 10Gb. Dave Handley

2 1

Re: [boost] Review of a safer memory management approach for C++?
by Bartlett, Roscoe A 04 Jun '10

04 Jun '10

Ingo, Mathias, and David, Sorry I missed your responses. It was not sent to me personally and I am only signed up for boost summaries. Is there some forum where I can present the approach described in: http://www.cs.sandia.gov/~rabartl/TeuchosMemoryManagementSAND.pdf and have an opportunity to discuss the various issues with interested parties and leaders in the C++ community? I am not sure the discussions on this mail list have adequately addressed the issues. My responses are below: > 6. Re: Review of a safer memory management approach for C++? > (Ingo Loehken) > 8. Re: Review of a safer memory management approach for C++? > (Mathias Gaunard) > 10. Re: Review of a safer memory management approach for C++? > (David Abrahams) > Message: 6 > Date: Thu, 27 May 2010 20:02:45 +0200 > From: Ingo Loehken <Ingo.Loehken(a)boc-eu.com> > To: boost(a)lists.boost.org > Subject: Re: [boost] Review of a safer memory management approach for > C++? > Message-ID: > <OF6B67C15A.1605A772-ONC1257730.00627F93-C1257730.00631B78@boc- > eu.com> > Content-Type: text/plain; charset="US-ASCII" > > if I understand "hard to reason about" in the right why : like there is > no > need for shared ownership at all, > this also means that there is no use for COM Programming - and of > course > there is. Raw resp. opaque > pointers have one huge advantage in productive environments : They work > as > perfect isolators, removing > any compile time dependencies. And from my point of view, compile time > is > what matters most, if you go > productive, otherwise you end up waiting for recompiles. Ingo, These are exactly the right points to be made. Many complex programs are going to involve sharing objects. We see this in many programming environments. Trying to "design" our way out of the sharing problem entirely will create more problems that it tries to solve. We need to separate the issues of sharing. One set of issues relates to change prorogation that will happen in any programming language (e.g. if objects A and B share object C and if A changes C what happens to B?). This is where we should be focusing our design attention. The other issue is silly memory management problems in C++ that other more modern languages like C#, Python, and Java don't suffer from so much. Objects stay around as long as they are needed. The goal of the approach described in: http://www.cs.sandia.gov/~rabartl/TeuchosMemoryManagementSAND.pdf is to as much as possible make the presence of sharing explicit so that we highlight the first type of "essential complexity" in programs but to minimize as much as possible the "accidental complexity" in trying to figure out when we need to actually delete an object and who's responsibility it is to delete the object. This is a very important issues and is discussed some in Section 6.1 in: http://www.cs.sandia.gov/~rabartl/TeuchosMemoryManagementSAND.pdf Cheers, - Ross > From: Mathias Gaunard <mathias.gaunard(a)ens-lyon.org> > To: boost(a)lists.boost.org > Date: 27.05.2010 14:59 > Subject: Re: [boost] Review of a safer memory management > approach > for C++? > Sent by: boost-bounces(a)lists.boost.org > > > Bartlett, Roscoe A wrote: > > Hello Boost developers, > > > > I am interested in finding one or more individuals who are > knowledgeable > about memory management in C++ (and especially of the reference- > counting > approach taken by boost::shared_ptr and boost::weak_ptr) to review an > idea > for a comprehensive approach to safer memory management in C++ that > encapsulates all raw C++ pointers in high-level code. > > Never use owning naked pointers and only use RAII (as is required for > exception-safe programming anyway) with exclusive ownership and no > aliasing, and you have no problems. [Bartlett, Roscoe A] Mathias, I agree but using raw pointers in any ubiquitous way is a problem. You can't detect dangling references and there are all of the other loosely typed problems described in Sections 2.1 and 2.2 in: http://www.cs.sandia.gov/~rabartl/TeuchosMemoryManagementSAND.pdf The issue is that it is easy to make mistakes with raw pointers related to ownership issues and if you use raw pointers, it is very difficult to detect them, even for tools like Valgrind and Purify. If no-one made mistakes, then we would all do well just to use void* and program in C. I want to eliminate undefined behavior in C++ programs in such a way that a properly configured static analysis tool could help to enforce. I believe that the approach described in http://www.cs.sandia.gov/~rabartl/TeuchosMemoryManagementSAND.pdf could be used with some static analysis tool to largely eliminate undefined behavior related to the usage of memory. There are lots of other areas of undefined behavior in C++ that this approach can't address (like bad type conversions of built-in types) but there are other idioms for addressing that. > Usage of shared_ptr should be an exception, not a widely deployed > solution to memory management issues. > Shared ownership is hard to reason about, and even if you use a similar > cycle-aware solution, cycles remain a real problem (they prevent > deterministic ordered destruction of objects, meaning they're only > applicable to certain classes of objects). [Bartlett, Roscoe A] No, if you share an object, you use a shared_ptr (or Teuchos::RCP), period. If sharing is the exception, then fine. Otherwise, don't play games with "I need shared_ptr here but I don't need it there because X, Y, and Z." We are never going to be able to produce programs that eliminate all sharing of objects, and we don't want to. Some types of programs are actually much simpler and more robust if you share some objects. For performance critical applications (e.g. the CSE domains I come from) you can't just willy nilly copy large objects and expect to achieve good performance. This is just not reality. > Message: 10 > Date: Thu, 27 May 2010 14:41:09 -0400 > From: David Abrahams <dave(a)boostpro.com> > To: boost(a)lists.boost.org > Subject: Re: [boost] Review of a safer memory management approach for > C++? > Message-ID: <m24ohtgpbu.wl%dave(a)boostpro.com> > Content-Type: text/plain; charset=US-ASCII > > At Thu, 27 May 2010 20:02:45 +0200, > Ingo Loehken wrote: > > > > if I understand "hard to reason about" in the right why : like there > > is no need for shared ownership at all, this also means that there > > is no use for COM Programming - and of course there is. > > I think you don't understand it the right way. Shared ownership (at > least in the presence of mutation) is hard to reason about because > seemingly-local modifications can have non-local effects. [Bartlett, Roscoe A] Yes, but let's focus on the "essential complexity" of object sharing and not get bogged down in the "accidental complexity" of memory management. The approaches described in http://www.cs.sandia.gov/~rabartl/TeuchosMemoryManagementSAND.pdf make explicit the "essential complexity" of object sharing while trying to eliminate the "accidental complexity" of memory management (see Section 6.1 in http://www.cs.sandia.gov/~rabartl/TeuchosMemoryManagementSAND.pdf) Trying to over design a program to avoid all shared ownership is what make C++ programming so unproductive and has all the negative consequences described in Section 1 in: http://www.cs.sandia.gov/~rabartl/TeuchosMemoryManagementSAND.pdf Designs with object sharing can be much less complex overall than designs with sharing. You just need decent tools to detect circular reference problems (and that is what the Teuchos::RCP class has). We have found that circular reference problems are fairly rare but when they occur they are a bear to track down. That is why I created all of the RCPNode tracing support in the Teuchos::RCP classes. Actually, we have found that non-deleted RCPNode objects are far more likely to occur due to other types of problems (e.g. calling abort(..), typical memory leaks, etc.) and in this role the RCPNode tracing machinery is a very cheap but effective version of memory checking that is automatically built into every debug-mode program and runs order of magnitude faster than tools like Valgrind and Purify. [Bartlett, Roscoe A] Cheers, - Ross Sorry again for not responding sooner.

3 2