- Boost - lists.boost.org

Re: [boost] [endian] swap_in_place use case
by Dave Handley 07 Jun '10

07 Jun '10

Please don't drop attributions. Vicente Botet wrote: > Dave Handley wrote: >> Memory map a network endian file. Swap_in_place. Use. >> >> You definitely don't want a copy in this case since your file could easily be >> very big. Think of the case where your file is huge (say 10Gb), you really >> don't want to perform a copy and swap since that puts your memory need at 20Gb >> instead of 10Gb. > Yes this could be a use case. I don't use to manage so bigs files. If I had > to work with I will never copy the whole file. But I'm not sure that I will > use a swap_in_place of the whole file. This could take too much time. I would > try to split the task to do on the whole file in smaller parts. > > What will you do with this big file, that makes the swap in place the best choice? The file could be many things. It could be a day of market data for a given exchange. It could be image data or video data that I'm going to perform image analysis on (maybe run a filter over it, or something similar). The file doesn't even have to be that big. If I was memory mapping a 10MB file and needed to swap it, I wouldn't want to use 20MB instead of 10MB. Pretty much anything I want to do to that file that involves looking at most or all of the data you would be much better off using swap in place instead of any copying swap implementation. Examples of the sorts of things that you might want to do to large files include running filters or normalisers over image or video files. I have lots of programs that have multiple threads constantly memory mapping files that range in size from relatively small to hundreds of MB or low numbers of GB. Given that memory allocation is a key component of the run time of these programs, they would run significantly slower if I had to allocate double the amount of memory. Don't forget, if you need the whole file to be swapped, then the fastest way to do it will be a swap in place of the whole file. I will reiterate something I said in an earlier post. If boost accepts an endian library which does not provide an efficient swap in place, I will be unable to use it. The library will end up in the list of boost libraries which are too inefficient to use in performance sensitive production code. Dave Handley

5 6

The bug sprint has started!
by Marshall Clow 07 Jun '10

07 Jun '10

As of 0800 Pacific time today, we had 989 open trac tickets. Info about the bug sprint can be found at <https://svn.boost.org/trac/boost/wiki/BugSprintJune2009> By ticket type: Bugs 613 Feature Requests 238 Patches 95 Tasks 27 Support Requests 15 Library Submissions 1 By milestone: To Be Determined 187 Boost 1.43.0 180 Boost 1.42.0 132 Boost 1.40.0 85 Boost 1.36.0 82 Boost 1.41.0 81 Boost 1.39.0 67 <none> 61 Boost 1.38.0 45 Boost 1.37.0 37 Boost 1.44.0 20 Website 1.X 10 Boost.Jam 4.0.0 1 Top 10 libraries: thread 76 date_time 55 Python 48 test 43 interprocess 40 iostreams 37 filesystem 36 mpl 36 quickbook 33 asio 31 Top 10 owners: anthonyw 75 djowel 71 dave 58 igaztanaga 58 az_sw_dude 53 vladimir_prus 49 bemandawes 46 rogeeff 43 agurtovoy 36 turkanis 36 Useful reports: https://svn.boost.org/trac/boost/report/1 -- full list of tickets https://svn.boost.org/trac/boost/report/18 -- ticket counts by owner https://svn.boost.org/trac/boost/report/19 -- ticket counts by component https://svn.boost.org/trac/boost/report/20 -- ticket counts by ticket type https://svn.boost.org/trac/boost/report/21 -- ticket counts by milestone

3 3

Re: [boost] Review of a safer memory management approach for C++?
by Bartlett, Roscoe A 07 Jun '10

07 Jun '10

Hello, This is the last email that I will write and respond to on this tread of emails about runtime vs. compile-time polymorphism (we have beet this dead horse about enough I think). I will summarize my arguments and the baulk responses from people. My perspective comes from having developed and maintained a templated package in Trilinos called Thyra that uses only the most basic type of templating; templating runtime abstract interfaces on the single Scalar type (i.e. double, float, complex<float>, complex<double>, etc). This is the most trivial and basic type of templating that you can possibly imagine (and was the type of templating that templates in C++ was supposed to support in the first place). Templating all of this code was not my first choice but in retrospect it was the right thing to do since you would have to duplicate all of the code (source and binary) multiple times (once for each scalar type) in order to support this variety of scalar types in C++ in a single program. However, this has not been easy by any stretch of the imagination. Now Trilinos is redoing is core linear algebra and linear solvers with the advent of the more heavily templated Tpetra package and will have similar issues. Up until recently, I was just having to compile and link Thyra and client code using the implicit template instantiation mechanism on the *wide* variety of compilers that we are contractually obligated to support including several different versions of GCC, Intel, Sun, PGI, IBM, Pathscale, and Visual C++ on a whole bunch of platforms (Linux, various flavors of Unix, Windows, specialty HPC systems, etc.). In many ways, I think that I have been the unfortunate first adopter of heavily templated code on many of these platforms. Eventually, I figured out how to portably and maintainability manage explicit template instantiation (see GCG 28 in http://www.cs.sandia.gov/~rabartl/ThyraCodingGuideLines.pdf) which has allowed me to port this templated code to several platforms where it would otherwise just segfault or crash the compilers (e.g. Sun and PGI). Going to carefully orchestrated explicit template instantiation has allowed me to recover a lot of the positive properties of non-templated code (i.e. fast recompile times, being able to compile just a single source files to work out compile problems, etc.). However, even with this most trivial type of templating (a single template argument), explicit instantiation is not automatic to support and significantly complicates the design and maintenance of the software. For new more heavily templated packages like Tpetra and the next generation of template software based on Tpetra in Trilinos that is being written, it is even more difficult and more complex to support explicit instantiation. If have fought the good fight for the use templated code in this non-ideal HPC environment for many years and I have earned to right to express my experiences and wisdom from the trenches on this topic. In return, the majority of the comments have basically been: a) Common response from list members: If have had trouble with templated code, it is because I don't know how to design or write software (just read some of the comments below and in some of the other email responses). My response: You can't get more simple templated code than a single Scalar template argument that is used in Thyra. I can't imagine a more basic use of templating and I have no clue how to design this "better". The only way to make this portable is to use explicit template instantiation which is not completely trivial to support, even for the most simplistic type of templating (i.e. just one Scalar template argument). b) Common response from list members: I should not bother supporting troublesome compilers like Sun, IBM, and PGI. My response: As I have explained many times, if our code can't support *all* of these platforms, then our customers can't use any of this code and we need to find another job. I don't know how much more direct I can state this. I have fought many battles against our customers who demand to use bad C++ compilers but we are going to have one or two bad compilers that we need to support for many years to come that throws a monkey wrench into the otherwise "perfect design" of the software we would like to write. You should see the emails that I get from people about "being in an ivory tower" when I complain about having to support non-ANSI C++ compilers. If some of my responses have been testy, it is because I am a little warn out with the above two general responses to problems that I have had with heavily templated code. Please don't just dismiss my opinions based on hard fought experience by assuming that I have no idea what I am doing. This entire thread of many emails and responses was sparked by my original request to review the approach described in: http://www.cs.sandia.gov/~rabartl/TeuchosMemoryManagementSAND.pdf which really has nothing to do with runtime vs. compile-time polymorphism. There has been a lot of responses but has anyone taken the time to even read at least the reduced table of contents described in the Preface? I know this is a long document at over 150 pages in the main text but the reduced table of contents is only something like 45 pages or so. A few selected responses to some other emails below ... > 1. Re: Review of a safer memory management approach for C++? > (Mathias Gaunard) > 3. Re: FW: Boost Digest, Vol 2933, Issue 1 (joel falcou) > ---------------------------------------------------------------------- > > Message: 1 > Date: Sat, 05 Jun 2010 17:59:24 +0100 > From: Mathias Gaunard <mathias.gaunard(a)ens-lyon.org> > To: boost(a)lists.boost.org > Subject: Re: [boost] Review of a safer memory management approach for > C++? > Message-ID: <hudvpc$76f$1(a)dough.gmane.org> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > ... > > > 2) Current compilers may not even be able to compile large amounts of > template code at all. There are compilers that we have to support > (e.g. Sun and IBM) that will segfault or return "internal compiler > error" when you give them too much template code. These same compilers > have no problem with virtual functions with standard OO. This is > reality. > > It's up to you whether you want to support those compilers or not. > I myself see little compelling reason to support the Sun compiler, > almost likewise for IBM, at least for new C++ projects. > > Balance your support for legacy compilers with your willingness to have > nice code. [Bartlett, Roscoe A] As explained above, I fight the battle all the time to avoid having to support such bad non-ANSI compilers but we will have to live with these problems for many years to come. If we can't support *all* of these platforms, then they can't use *any* of the code that does not port to all of these platforms, period. This is not a joke. I suspect that Trilinos and other HPC projects are not the only software projects that have to live this reality. > > 3) Templated programs can result in massive object-code bloat over > equivalent runtime OO programs. If you are really sloppy with > templates and use it for everything, you can easily double, triple, or > more (10x) the size of the object code and executables (see Item 44 in > "Effective C++, 3ed edition"). > > They can, but they don't have to. > If your template function is not a good inlining candidate, is likely > to > get instantiated with many different types and and if making it > runtime-generic has sufficiently low costs, then you should consider > doing it. [Bartlett, Roscoe A] 99% of Thyra-based code should not ever be inlined (see description of Thyra above or just go look at the Thyra Website). > Note, however, that the size costs are usually grossly exaggerated. > I've > seen people use advanced template C++ code on microcontrollers (the > kind > with a few kilobytes of memory) just fine. [Bartlett, Roscoe A] If you templated the algorithms instead of using Thyra interfaces, you would have new and different object code for every concrete set of operator/vector subclasses. Just in Trilinos this would result in at least a 3x increase in the size of the object code. There is no way around this. > > As long as people keep suggesting that we template all of our codes > for every little reason > > No one is suggesting this. > Make your code only as generic as your algorithms intrinsically are, no > more. > > Algorithms are what lay at the center of generic programming, not > objects. This is quite a different view from OOP. [Bartlett, Roscoe A] Using Thyra as the example, if you throw away the abstract interfaces, then 99% of your algorithmic code would be templated in order to achieve (compile-time) polymorphic behavior. This is a very simple issue to analyze. > > Thinking that we are going to drop runtime polymorphism and use > exclusively compile-time polymorphism for everything is just fantasy > land. > > Compile-time polymorphism is better, and you can build runtime > polymorphism on top of it with type erasure. > > So why avoid compile-time polymorphism when you can do it? [Bartlett, Roscoe A] The only way to manage large amounts of templated code is to explicitly instantiate it which is not trivial to code and maintain (again, see above). > ------------------------------ > > Message: 3 > Date: Sat, 05 Jun 2010 19:08:55 +0200 > From: joel falcou <joel.falcou(a)lri.fr> > To: boost(a)lists.boost.org > Subject: Re: [boost] FW: Boost Digest, Vol 2933, Issue 1 > Message-ID: <4C0A84A7.7060002(a)lri.fr> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > ... > > And I can tell you that, as a consultant and now as an assistant > professor involved in variosu HPC project, the quality of code I saw in > laboratory is usually sub-par. [Bartlett, Roscoe A] What is this supposed to mean? > > The size of object code does have some impact, especially some HPC > machines. Again, please teach us all how to do this because we > apparently don't know what we are doing. > > > You can also learn to reply without actually barking at people. It > helps > having a sane conversation. [Bartlett, Roscoe A] As described at the top, when the common response to problems is that "you don't know how to design software" one has almost no choice but to respond in such a way. Again, the templating in Thyra could not be more simple. > > The compilers segfault or return internal compiler errors or take > hours to compile if they do finish at all in some cases. See above. > Reality!!!!!!! > > > Reality of sucky compilers from 2 century ago. It's lkike blaming > people > using new cars with biofuel whiel comapring them to Ford T. [Bartlett, Roscoe A] As described at the very top, I fight the battle against bad C++ compilers constantly but we are going to have to live with some bad compilers for many years to come. What do you expect us to do? Should I just quit my job as a protest against bad C++ compilers? > > My argument is that current accepted ways of writing OO code in C++ > make it too hard to write correct well designed programs and an > affordable cost. My argument is that the Teuchos MM approach described > in: > > > > http://www.cs.sandia.gov/~rabartl/TeuchosMemoryManagementSAND.pdf > > > > together with static analysis tools to help enforce the idioms (and > keep raw pointers encapsulated) can largely eliminate undefined > behavior in C++ programs due to the usage of memory and also yield > programs that are more self-documenting at the same time. > > > > > Well, you better be writing JAVA or OCaml ... [Bartlett, Roscoe A] Not true. As described in Section 6.1 in: http://www.cs.sandia.gov/~rabartl/TeuchosMemoryManagementSAND.pdf the idioms using the Teuchos MM classes can yield better self documenting code than you can get with languages like Java and Python. Did you bother to read the document? Yes I know it is very long but I think there is some worthwhile material contained in it that should be given a far shake and not just dismissed by uninformed email responses. (By uninformed, I mean you likely did not ready the document, not that you are fundamentally uninformed). - Ross

2 1

[thread] Customizing barrier for improved performance
by Belcourt, Kenneth 06 Jun '10

06 Jun '10

Hi, I notice that the thread barrier class is fairly large (128 bytes on Darwin with Intel 11.1). private: mutex m_mutex; condition_variable m_cond; unsigned int m_threshold; unsigned int m_count; unsigned int m_generation; and sort of slow for my application (parallel iterative solvers of sparse linear systems). Many iterative algorithms have both serial and parallel sections during a single iteration and, for larger algorithms, this can result in numerous (order 10 or so) rendezvous points during each iteration. During cursory testing I've found that a barrier implemented with atomics is a bit faster than a mutex based barrier (though I recognize that an atomic spin-based implementation can potentially hang if running on a single Intel core with hyper- threading enabled). I've attached a simple atomic based implementation built on Intel tbb atomic though it's easily convertible to boost.atomic when the time comes. This implementation just ping-pongs a counter alternating between incrementing and decrementing the counter each time it's called. Does anyone know if there's plans to extend barrier so that a user could select a different implementation (like an atomic based one)? For some applications this could be a very useful extension. Thanks. -- Noel

1 0

[filesystem] Version 3 of Boost.Filesystem added to trunk
by Beman Dawes 06 Jun '10

06 Jun '10

Because version 3 will break some user code, both v2 and v3 will be shipped for several releases. For 1.44, the default is v2 and the user has to explicitly switch to v3. I'd really appreciate it if some Boosters could give v3 a try on your own code that uses Boost.Filesystem. If you already have trunk on your machine, just run "svn up" on it. Otherwise, checkout a copy: mkdir boost-trunk svn co http://svn.boost.org/svn/boost/trunk boost-trunk If you prefer, "svn export http://svn.boost.org/svn/boost/trunk boost-trunk" should also work. Then read boost-trunk/libs/filesystem/dual_versions.html, and follow the instructions there. Then please let me know your results: * Are the instructions for changing to version 3 sufficient? They are pretty minimal, but I'm assuming those doing this have a high level of background knowledge. * Did you have any problem changing to version 3? * Did you have problems compiling your code against version 3? * If there were compile errors, do you have any ideas about how to mitigate the problem? * Did you have any problems testing or running your code with version 3? * Any other comments or suggestions? Thanks, --Beman

8 28

Bug Sprint daily update for Sunday, June 6th
by Marshall Clow 06 Jun '10

06 Jun '10

As of 0800 Pacific time today, we had 988 open trac tickets. This is down by one from yesterday. However, there was lots of activity on the Trac yesterday: There were 4 tickets added There were modifications to 24 tickets. (ok, some were the same ticket more than once, but still...) There were 5 tickets closed. Daniel James did a lot of cleanup on the Quickbook tickets. Vicente Botet did an analysis of many of the Boost.Thread tickets and posted it to the list. Open tickets by type: Bugs 611 (-2) Feature Requests 239 (+1) Patches 95 Tasks 27 Support Requests 15 Library Submissions 1 Open tickets buy milestone: To Be Determined 186 (-1) Boost 1.43.0 181 (+1) Boost 1.42.0 132 Boost 1.40.0 85 Boost 1.36.0 82 Boost 1.41.0 81 Boost 1.39.0 67 <none> 60 (-1) Boost 1.38.0 45 Boost 1.37.0 37 Boost 1.44.0 20 Website 1.X 10 Boost.Jam 4.0.0 1 Top 10 libraries: thread 76 date_time 54 (-1) Python 48 test 43 interprocess 40 iostreams 37 filesystem 36 mpl 36 quickbook 32 (-1) asio 31 Top 10 owners: anthonyw 75 djowel 66 (-5) dave 58 igaztanaga 58 az_sw_dude 52 (-1) vladimir_prus 49 bemandawes 46 rogeeff 43 agurtovoy 36 turkanis 36 -- Marshall

1 0

[phoenix] Result type & compatibility with result_of
by Andrey Semashev 06 Jun '10

06 Jun '10

Hi, Is there a rationale of Phoenix2 being not compatible with STL functors and result_of? I'm trying to bind an STL functor and it fails to compile.

2 1

[Proto] Can't statement like expression be part of proto
by joel falcou 05 Jun '10

05 Jun '10

after digetsing all the contents of eric boost'con talk i came to the conclusion that epxression-tempalte enabled statement may be part of proto like operators. One can see the ability of having if or for as a real expression geenrator for any EDSL as actaully a feature. What may prevent if_, for_ and such to be first class, EDSL agnostic citizen fo proto ?

2 1

[polygon] tutorial done and library ready for release
by Simonson, Lucanus J 05 Jun '10

05 Jun '10

I have just added a tutorial to the documentation. The tutorial walks the reader through the requirements, design and implementation of a toy layout versus schematic comparision application. I would like feedback from the community on whether this is what you are looking for in a tutorial. Also I'd like to check with my review manager, Fernando, if this tutorial satisfies the acceptance requirement that I add a tutorial to the documentation. The tutorial can be found here: https://svn.boost.org/svn/boost/sandbox/gtl/doc/gtl_tutorial.htm At this stage I think I have satisfied the acceptance requirements from the review and would like to coordinate merging the library onto the trunk and release branch to get into the next release of boost. A little guidance from the release manager would be appreciated. Thanks, Luke

9 28

FW: Boost Digest, Vol 2933, Issue 1
by Bartlett, Roscoe A 05 Jun '10

05 Jun '10

Joel, > 13. Re: Review of a safer memory management approach for C++? > (joel falcou) > ------------------------------ > > Message: 13 > Date: Sat, 05 Jun 2010 16:40:01 +0200 > From: joel falcou <joel.falcou(a)lri.fr> > To: boost(a)lists.boost.org > Subject: Re: [boost] Review of a safer memory management approach for > C++? > Message-ID: <4C0A61C1.3020804(a)lri.fr> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > Bartlett, Roscoe A wrote: > > 1) Well written OO C++ software builds and (more importantly) > rebuilds *much* faster than heavily templated code. This massively > speeds up the development cycle. The difference can be orders of > magnitude of time savings in recompile/relink times. I have seen cases > where carefully using runtime OO can result in recompile/relink cycles > take less than 20 seconds where using all template code would take 20 > minutes. I am *not* exaggerating here at all. > > > You have the right to write proper, fast to compile template code. > Technics and tools exists for this. And if your tempalte takes 20mn to > compile, your template design is wrong ... [Bartlett, Roscoe A] Nonsense. Apparently myself and no-one else in CSE or any other domain knows the right way to write templated code. Apparently all the books I have read on C++ and template programming (see http://www.cs.sandia.gov/~rabartl/readingList.html#_B%29_C++_Books) have not taught me the *right* way to write template C++ code. If it is that hard to write "correct" template C++ code then C++ should die a slow and painful death because no-one will be able to use it correctly. Oh, would you please teach us? > In our own HPC template base > code, everything compiles under 20s and only some exhaustive unti test > that instanciate all and every type/functiosn took up to 5mn. > > 2) Current compilers may not even be able to compile large amounts of > template code at all. There are compilers that we have to support > (e.g. Sun and IBM) that will segfault or return "internal compiler > error" when you give them too much template code. These same compilers > have no problem with virtual functions with standard OO. This is > reality. > > > Sun and IBM are all but references on the point of tempalte handling > ... > gcc, MSVC, ICC and even clang don't choke on most templated code ... > What about stopping working with compilers from before the industrial > revolution ? [Bartlett, Roscoe A] We don't get to pick the platforms that we have to support. If you supply code to ASC customers you will compile on their platforms, including Sun, IBM, and PGI. In fact, this is one reason why we can't have any mandatory dependencies on boost because there have been portability problems with boost on some of these platforms. Things have gotten a lot better with C++ compilers that we have to use in the last 9 years since I have been at Sandia but it is just not true that implicit instantiation with template code will compile just as fast as non-template code. There are fundamental arguments to refute that. Compilers could be much more crafty with templated code than they currently are but they are not (yet) and never will be with some compilers (i.e. Sun, IBM, PGI, etc.). > > 3) Templated programs can result in massive object-code bloat over > equivalent runtime OO programs. If you are really sloppy with > templates and use it for everything, you can easily double, triple, or > more (10x) the size of the object code and executables (see Item 44 in > "Effective C++, 3ed edition"). > > > > > same than 1/ learn to do it right ... And well, since when gigabyte > started to be costly ? [Bartlett, Roscoe A] The size of object code does have some impact, especially some HPC machines. Again, please teach us all how to do this because we apparently don't know what we are doing. > > 5) Well written OO C++ software allows for significant runtime > flexibility. If you use dynamic libraries, for instance, you can add > new implementations of objects without even having to relink customer > executables. Also, you can fix many types of bugs in updated shared > libraries without having to recompile/relink customer executables. In > a large-scale software setting, this can be very helpful. > > > That's maybe th eonyl poitn where I agree with you. > > > *********************************************************************** > ********************* > > *** You can't portably compile and link large C++ programs that > template everything!!!!! *** > > > *********************************************************************** > ********************* > > > Why ? [Bartlett, Roscoe A] The compilers segfault or return internal compiler errors or take hours to compile if they do finish at all in some cases. See above. Reality!!!!!!! > > As long as people keep suggesting that we template all of our codes > for every little reason, I am going to keep reminding people of the > realities of template code in C++. If you can't respond to the above > points (especially #1 and #2) then there is little point in continuing > the discussion. Thinking that we are going to drop runtime > polymorphism and use exclusively compile-time polymorphism for > everything is just fantasy land. > > > Nobody told you to put template everywhere. Same we don't want to put > dynamic polymorphism everywhere. What about learnign to sue the proper > tool for the proper task ? [Bartlett, Roscoe A] Finally we agree. Runtime polymorphism with OO has its place as does templated code with compile-time polymorphism. I was never arguing that template methods should never be used (because I use them all the time, much more than most C++ programmers) but other people did seem to be arguing to never use runtime OO with virtual functions (which is just bad advice). My argument is that current accepted ways of writing OO code in C++ make it too hard to write correct well designed programs and an affordable cost. My argument is that the Teuchos MM approach described in: http://www.cs.sandia.gov/~rabartl/TeuchosMemoryManagementSAND.pdf together with static analysis tools to help enforce the idioms (and keep raw pointers encapsulated) can largely eliminate undefined behavior in C++ programs due to the usage of memory and also yield programs that are more self-documenting at the same time. That is it. - Ross

2 1