June 2010 - Boost - lists.boost.org

[trac] "Building Boost" vs. "build"
by Vladimir Prus 07 Jun '10

07 Jun '10

It looks like somebody went via Trac and moved some issues from the "Building Boost" component to "build" component. See, for example: https://svn.boost.org/trac/boost/ticket/4147 I certainly appreciate help with triaging issues, but in general, such change seems wrong -- "Building Boost" is for issues affecting exactly that -- building boost, while the 'build' component contain issues that are not necessary affecting Boost C++ Libraries at all. Unfortunately, the changes are all anonymous, so it would be great if whoever made them speak up, and discuss the rationale. Thanks, -- Vladimir Prus CodeSourcery vladimir(a)codesourcery.com (650) 331-3385 x722

4 5

Active Tickets report and accepted info
by vicente.botet 07 Jun '10

07 Jun '10

Hi, in https://svn.boost.org/trac/boost/report/1 I see the following: "If a ticket has been accepted, a '*' is appended after the owner's name" Is there an error on the report or the maintainers do not use to accept the tickets? I find this information very useful, if it states that the maintainer accept to take care of this ticket. Best, _____________________ Vicente Juan Botet Escribá http://viboes.blogspot.com/

5 7

Re: [boost] Review of a safer memory management approach for C++?
by Bartlett, Roscoe A 07 Jun '10

07 Jun '10

David, Much more discussion of this topic over email is likely not very useful at this point but I will make some very brief responses below. > 14. Re: FW: Boost Digest, Vol 2929, Issue 4 (David Abrahams) > ------------------------------ > > Message: 14 > Date: Fri, 04 Jun 2010 09:42:36 -0400 > From: David Abrahams <dave(a)boostpro.com> > To: boost(a)lists.boost.org > Subject: Re: [boost] FW: Boost Digest, Vol 2929, Issue 4 > Message-ID: <m2hbliewxf.wl%dave(a)boostpro.com> > Content-Type: text/plain; charset=UTF-8 > > At Thu, 3 Jun 2010 17:29:15 -0600, > Bartlett, Roscoe A wrote: > > > > David, > > > > I have some responses below ... > > > > > Message: 13 > > > Date: Tue, 01 Jun 2010 16:16:25 -0400 > > > From: David Abrahams <dave(a)boostpro.com> > > > > > > I'm all for making essential complexity obvious and eliminating > > > accidental complexity, but I'm not sure I see a compelling example > > > that Teuchos does that, at least not when compared with what I > > > consider normal state-of-the-art C++ programming, which is mostly > > > value-based, dynamic allocations are rare, and those that occur are > > > immediately managed, e.g. by appropriate smart pointers. > > > > [Bartlett, Roscoe A] > > > > I don't want to get into an argument about the "proper" way to use > > C++ here (but I will a little below). In Item 1 of "Effective C++, > > 3rd Edition", Scott Meyers identifies four different major > > programming paradigms that are supposed to be supported by C++: > > > > 1) C: raw pointers, low-level built-in datatypes (int, double, > > etc.), statements, blocks, functions etc. > > > > 2) Object-oriented C++: Inheritance, virtual functions, dynamic > > allocation, RTTI, etc. (i.e. runtime polymorphism) > > > > 3) Template C++: Typical generic programming, template > > metaprogramming, etc. (i.e. compile-time polymorphism) > > > > 4) STL: Containers, iterators, algorithms, etc. and code like this > > (not just using STL) (closely related to #3 Template C++). > > Yes, I'm aware of that way of dividing up the picture. > > > Of course any well written large C++ program is a mixture of all > > four of the above (and more). > > I don't take it for granted that any well-written large C++ program > will use any given paradigm. > > > What you seem to be arguing above is that most good C++ software > > will use templates in #3 and #4 but almost none of #2 > > object-oriented C++. > > Did I say anything about templates? No. > > I _will_ argue that OOP has costs to software complexity that are > often under-appreciated. [Bartlett, Roscoe A] Any approach can be misused, including anything you might be trying to advocate (but I am not 100% sure what that is exactly). > > Note that C++ was first designed in the early > > 80s primarily to support #2 object-oriented C++. I can see the > > reason for this bias for #3 and #4 given the nature of most of the > > Boost libraries and you can have that opinion if you would like. > > I don't, so please read what I wrote and try to address that rather > than to your preconceived notions of my ?bias.? [Bartlett, Roscoe A] I am not trying to put words in your mouth. I am just trying to pin down your argument/position. You can say that I have a "bias" for good software if you would like :-) Seriously, I am sorry if I misinterpreted you position with the position of others on the boost list who were arguing for a more template-heavy approach. I guess that you are arguing for more of a stack-based approach (see below). > > I would argue that the current accepted approaches to writing OO > > programs in C++ (i.e. #2 OO C++ features) make it too easy to create > > programs with undefined behavior (i.e. segfaults or worse) and as a > > result, the paranoia of undefined behavior in C++, memory leaks > > etc. lead to lots of bad programming practices and designs (see > > Section 1 in > > http://www.cs.sandia.gov/~rabartl/TeuchosMemoryManagementSAND.pdf) > > Seriously? Are you saying that a disciplined application of > shared_ptr, weak_ptr, new_shared, et. al when dynamic allocations are > called for is insufficient for most OO programs? [Bartlett, Roscoe A] As argued in the Teuchos MM document http://www.cs.sandia.gov/~rabartl/TeuchosMemoryManagementSAND.pdf these boost classes are a good start but they are *not* sufficient to build a safe and effective memory management system that encapsulated *all* raw pointers. What is missing for single objects, as described in Section 5.14 and other sections, is the Teuchos::Ptr class. Without integrating a class like Ptr in with the reference-counted pointer classes, you can't build a safe and effective memory management system in C++ that comes closer to eliminating undefined behavior. Also, as described in Section 5.9.2, separating weak_ptr from shared_ptr is not ideal from a design perspective since it reduces the generality and reusability of software (see the arguments for this in the document). Did you read at least the sections called out in the reduced table of contents given in the Preface? I think these sections should address all of these questions (and more). As for arrays, the current software in C++0x and boost is less sufficient (again, see Section 5.14 and the sections it refers to). What is in boost and C++0x is a good start but it has not gone far enough (i.e. the encapsulation of *all* raw pointers in *all* code, except where non-compliant code must be called of course). > > > > Trying to over design a program to avoid all shared ownership is > > > > what make C++ programming so unproductive and has all the > negative > > > > consequences described in Section 1 in: > > > > > > > > > http://www.cs.sandia.gov/~rabartl/TeuchosMemoryManagementSAND.pdf > > > > > > > > Designs with object sharing can be much less complex overall than > > > > designs without sharing. You just need decent tools to detect > circular > > > > reference problems (and that is what the Teuchos::RCP class has). > > > > > > Well, I fundamentally disagree with all of the above. Overuse of > > > runtime polymorphism, thus dynamic allocation, and thus shared > > > ownership (it's almost an inevitable progression) is one of the > things > > > that has made C++ programming unproductive, and those people I know > > > who most zealously avoid it tend to be more productive than > everyone > > > else. IMO. > > > > [Bartlett, Roscoe A] > > > > When runtime performance or other issues related to dynamic > > allocations and classic OO are not a problem, classic OO C++ > > programs using runtime polymorphism are typically superior to highly > > templated C++ programs using static polymorphism > > Straw man; I'm not arguing for ?templated C++ programs using static > polymorphism.? [Bartlett, Roscoe A] Okay, the runtime vs. compile-time polymorphism debate is another issue (but a very important one). Are you instead arguing for a static and stack-based approach to programming? I haven see people try this and it leads to very inflexible software that is hard to maintain, reuse, and achieve optimal performance. One particularly good (or bad) example of this approach that I have seen is given in: http://www.trip.caam.rice.edu/software/rvldoc/rvl.html If you look deep in the RVL code, you will see a few Gotchas that destroy the true safely that is supposed to be afforded by the approach. Actually, ironically, they use stack-based object to try to avoid memory management problems (but not sufficiently) but yet at the same time, they rely on other types of mutable shared objects with implicit updates that make my very unconformable. > > > In my experience, designs with object sharing tend to increase > > > complexity in all kinds of ways beyond the memory management issues > > > that you describe. Perhaps the most important way is, they tend to > > > obscure the overall computation being performed by hiding it in > > > networks of interacting objects. > > > > [Bartlett, Roscoe A] > > > > This is not going to magically change by templating everything. > > Straw man; I'm not arguing for ?templating everything.? > > > If a lot of objects collaborate together to perform computations, it > > does not matter if you use static or runtime polymorphism; the code > > will be hard to figure out and you will need print statements and/or > > run it in a debugger to see what is really happening. > > Yes, and an ?OO mindset? has tended to lead to partitioning systems so > that ?a lot of objects collaborate together to perform computations,? > which tends to obscure the computation performed. [Bartlett, Roscoe A] Yes but good OO design tries to minimize the web of object dependences. There are lots of good references on the need and methods for reducing the endless web of object dependencies (see http://www.cs.sandia.gov/~rabartl/readingList.html) > > > I would like to see an example of a design with shared object > > > ownership that is ?much less complex? than all other functionally > > > equivalent designs that eschew shared ownership. I'm not saying > such > > > applications don't exist, but IME they are rare, and a concrete > > > example would help to bring this whole discussion back to earth. > > > > [Bartlett, Roscoe A] > > > > One design where shared mutable objects likely makes things overall > > simpler and more natural is the PDE expression mechanism in the > > Sundance code (http://www.math.ttu.edu/~klong/Sundance/html/) If > > the clients all expect remote updates of shared objects, then things > > work great. > > I'm sorry, your reference isn't specific enough for me to see the > design piece you're referring to. [Bartlett, Roscoe A] It would take a significant effort to pin-point exactly the approach being used in the Sundance expression mechanism and then to describe an alternative design that does not rely on the kind of object sharing. I think that this example in Sundance is the exception and not the rule. In general, I don't like lots for mutable shared objects for the same reasons that you state. However, there are cases where using shared mutable objects is the best design to solve a given problem. As for dynamic vs. stack-based programming, in summary, if you like the approach used in: http://www.trip.caam.rice.edu/software/rvldoc/rvl.html which tries to to make *every* object a stack-based object, then you and I can safely agree to disagree about classic OO vs. purely stack-based programming. Sorry I misinterpreted your position (or projected other peoples positions on you) about the runtime vs. compile-time polymorphism debate. Cheers, - Ross

4 7

FW: Boost Digest, Vol 2929, Issue 4
by Bartlett, Roscoe A 07 Jun '10

07 Jun '10

David, I have some responses below ... > ------------------------------ > > Message: 13 > Date: Tue, 01 Jun 2010 16:16:25 -0400 > From: David Abrahams <dave(a)boostpro.com> > To: boost(a)lists.boost.org > Cc: "mathias.gaunard(a)ens-lyon.org" <mathias.gaunard(a)ens-lyon.org> > Subject: Re: [boost] Review of a safer memory management approach for > C++? > Message-ID: <m2hblmcxuu.wl%dave(a)boostpro.com> > Content-Type: text/plain; charset=UTF-8 > > ... > > > > At Thu, 27 May 2010 20:02:45 +0200, > > > Ingo Loehken wrote: > > > > > > > > if I understand "hard to reason about" in the right why : like > there > > > > is no need for shared ownership at all, this also means that > there > > > > is no use for COM Programming - and of course there is. > > > > > > I think you don't understand it the right way. Shared ownership > (at > > > least in the presence of mutation) is hard to reason about because > > > seemingly-local modifications can have non-local effects. > > > > [Bartlett, Roscoe A] > > > > Yes, but let's focus on the "essential complexity" of object sharing > > Okay. > > > and not get bogged down in the "accidental complexity" of memory > > management. The approaches described in > > > > http://www.cs.sandia.gov/~rabartl/TeuchosMemoryManagementSAND.pdf > > > > make explicit the "essential complexity" of object sharing while > > trying to eliminate the "accidental complexity" of memory management > > (see Section 6.1 in > > http://www.cs.sandia.gov/~rabartl/TeuchosMemoryManagementSAND.pdf) > > I'm all for making essential complexity obvious and eliminating > accidental complexity, but I'm not sure I see a compelling example > that Teuchos does that, at least not when compared with what I > consider normal state-of-the-art C++ programming, which is mostly > value-based, dynamic allocations are rare, and those that occur are > immediately managed, e.g. by appropriate smart pointers. [Bartlett, Roscoe A] I don't want to get into an argument about the "proper" way to use C++ here (but I will a little below). In Item 1 of "Effective C++, 3rd Edition", Scott Meyers identifies four different major programming paradigms that are supposed to be supported by C++: 1) C: raw pointers, low-level built-in datatypes (int, double, etc.), statements, blocks, functions etc. 2) Object-oriented C++: Inheritance, virtual functions, dynamic allocation, RTTI, etc. (i.e. runtime polymorphism) 3) Template C++: Typical generic programming, template metaprogramming, etc. (i.e. compile-time polymorphism) 4) STL: Containers, iterators, algorithms, etc. and code like this (not just using STL) (closely related to #3 Template C++). Of course any well written large C++ program is a mixture of all four of the above (and more). What you seem to be arguing above is that most good C++ software will use templates in #3 and #4 but almost none of #2 object-oriented C++. Note that C++ was first designed in the early 80s primarily to support #2 object-oriented C++. I can see the reason for this bias for #3 and #4 given the nature of most of the Boost libraries and you can have that opinion if you would like. However, there are clearly types of programming problems that call for more of #2 and less of #3 and #4 in the overall design. Languages like Java and Python would be typically used to write safe object-oriented programs in a productive environment (free of complications of dynamic memory allocation and most undefined behavior that is so easy to expose with current C++ approaches). I will argue below that more C++ programs are better served by using more of #2 object-oriented C++ and to a lesser extent template-based methods in #3 and #4 (except where it is called for in lower-level code). Instead of arguing if we should be using #2 object-oriented C++ (runtime polymorphism) or instead prefer #3 template programming (static polymorphism), let's assume that we want C++ to do a good job of supporting #2 object-oriented C++ if we decide to use it. Therefore, the real question is if C++ is indeed supposed to support #2 object-oriented programming, does C++ it support it well enough so that I would choose to use C++ instead of Java or Python (assuming my programming team knows those languages)? By all reasonable measures, writing object-oriented programs in Java or Python is more productive because of garbage collection and the elimination of undefined behavior that is so common in C++ programs (I can find lots of references for these opinions and even data from studies to support this claim). However, to achieve performance in CSE software, I have to write some code in C/C++ or some other appropriate language. No-one has been able to demonstrate sufficient performance for CSE codes written entirely in Java or Python. Also, many of the "safer" and "more productive" OO languages (e.g. Java and Python) are not even available on some high-end HPC platforms. Even if they were, mixed language programming has lots of problems that discourages wide-spread use. That leaves C++ as the only fully portable viable language for high-performance CSE (sorry Fortran programmers). I would argue that the current accepted approaches to writing OO programs in C++ (i.e. #2 OO C++ features) make it too easy to create programs with undefined behavior (i.e. segfaults or worse) and as a result, the paranoia of undefined behavior in C++, memory leaks etc. lead to lots of bad programming practices and designs (see Section 1 in http://www.cs.sandia.gov/~rabartl/TeuchosMemoryManagementSAND.pdf) What the approach described in the Teuchos MM report tries to do is to create a set of classes and idioms that can allow C++ to be used to safely and productively develop #2 OO C++ software closer to the safety and productively that you can get with languages like Java and Python. However, there is still too much "accidental complexity" in C++ to have it ever fully compete with productivity of writing in Java and Python for basic OO features (IMO). In any case, C++ offers other advantages over any other language that would have you choose it over these other languages. > > Trying to over design a program to avoid all shared ownership is > > what make C++ programming so unproductive and has all the negative > > consequences described in Section 1 in: > > > > http://www.cs.sandia.gov/~rabartl/TeuchosMemoryManagementSAND.pdf > > > > Designs with object sharing can be much less complex overall than > > designs without sharing. You just need decent tools to detect circular > > reference problems (and that is what the Teuchos::RCP class has). > > Well, I fundamentally disagree with all of the above. Overuse of > runtime polymorphism, thus dynamic allocation, and thus shared > ownership (it's almost an inevitable progression) is one of the things > that has made C++ programming unproductive, and those people I know > who most zealously avoid it tend to be more productive than everyone > else. IMO. [Bartlett, Roscoe A] When runtime performance or other issues related to dynamic allocations and classic OO are not a problem, classic OO C++ programs using runtime polymorphism are typically superior to highly templated C++ programs using static polymorphism for the following (indisputable) reasons: 1) Well written OO C++ software builds and (more importantly) rebuilds *much* faster then heavily template code. This massively speeds up the development cycle. The difference can be orders of magnitude of time savings in recompile/relink times. I have seen cases where carefully using runtime OO can result in recompile/relink cycles take less than 20 seconds where using all template code would take 20 minutes. I am *not* exaggerating here at all. 2) Templated programs can result in massive object-code bloat over equivalent runtime OO programs. If you are really sloppy with templates and use it for everything, you can easily double, triple, or more the size of the object code and executables (see Item 44 in "Effective C++, 3ed edition"). 3) The error messages you get when mistakes are made when using interfaces and virtual functions are far superior to the mess that most compliers put out when you make a mistake with implicit compile-time template interfaces. This is a huge issue. Runtime errors, as opposed to cryptic compile-time errors, can be handled by throwing exceptions with very good error messages that programmers can 100% control the quality of. Yes, of course you prefer compile-time errors to runtime errors all things being equal, and am not arguing with that. This is generally not true of template programming (but there are some tricks you can use to improve compile-time error messages, but not perfectly). For example, if you make a mistake with STL iterators/algorithms you can get very cryptic error messages that dumfound most (non-expert and expert) C++ programmers. 4) Well written OO C++ software allows for significant runtime flexibility. If you use dynamic libraries, for instance, you can add new implementations of objects without even having to relink customer executables. Also, you can fix many types of bugs in updated shared libraries without having to recompile/relink customer executables. In a large-scale software setting, this can be very helpful. Believe me, I see the use of template C++ code with lots of static typing and lots of local logic and control but it is not the end all and will never replace classic OO C++ for most high-level designs and software architecture. I challenge anyone to refute this. C++ compilers will need to become many orders of magnitude faster that what they are today for this fact to change (but then the software will just grow that much larger and erase the compiler speed improvements). > In my experience, designs with object sharing tend to increase > complexity in all kinds of ways beyond the memory management issues > that you describe. Perhaps the most important way is, they tend to > obscure the overall computation being performed by hiding it in > networks of interacting objects. [Bartlett, Roscoe A] This is not going to magically change by templating everything. If a lot of objects collaborate together to perform computations, it does not matter if you use static or runtime polymorphism; the code will be hard to figure out and you will need print statements and/or run it in a debugger to see what is really happening. > I would like to see an example of a design with shared object > ownership that is ?much less complex? than all other functionally > equivalent designs that eschew shared ownership. I'm not saying such > applications don't exist, but IME they are rare, and a concrete > example would help to bring this whole discussion back to earth. [Bartlett, Roscoe A] One design where shared mutable objects likely makes things overall simpler and more natural is the PDE expression mechanism in the Sundance code (http://www.math.ttu.edu/~klong/Sundance/html/) If the clients all expect remote updates of shared objects, then things work great. The problem is when an object is changed by another client and you don't expect it to change (i.e. the classic problem of change propagation). As for sharing of objects, it is critical for many types of software in the Computational Science and Engineering (CSE) domain. You have to share large objects in order to cut down on runtime and storage overhead. As one example, consider the Petra object model with the Trilinos packages Epetra and Tpetra (http://trilinos.sandia.gov/) In the Petra object model, you have objects with significant storage overhead like Maps and Graphs that are shared between many different objects like Vectors, MultiVectors, and Matrices. Yes, you could create deep copies of Maps and Graphs but that would waste a large amount of storage. The bottom line is that by sharing large objects like Maps and Graphs instead of making large numbers of copies, we can fit and solve much larger simulation problems on a computer than we could otherwise. You could template and statically allocate everything and you will never get around the fundamental need to share the data contained in large objects in these types of applications. Note, however, that Maps in Epetra/Tpetra are shared as const immutable objects. That means that no-one can change a Map after it is first created. Therefore, all of the problems with unexpected updates goes away (as you mention above). However, the shared Maps must go away when they are not needed anymore and that is what the Teuchos::RCP class makes easy and robust. Under the covers, Tpetra uses Tpetra::ArrayRCP for similar purposes and much more. The situation with Graphs is different and issues of change propagation of shared objects are still present in some cases. Come to think of it, most of the object sharing that used in the CSE software that I use and write mostly just share const immutable objects so problems of change propagation mostly goes away. However, there are some important use cases where all of the clients are not holding RCPs to const objects and the problem of change propagation remains. Again, to save in significant runtime and storage overhead, we can't just create deep copies of all of these objects. Templating and purely stack-based programs are not going to solve that problem. The Teuchos MM approach, I believe, largely solves the memory management problems with dynamic memory allocation and object sharing while still yielding very high performance and mostly eliminating undefined behavior associated with the incorrect usage of memory. The most significant contribution of the Teuchos MM approach is the presents of Teuchos::Ptr and Teuchos::ArrayView and how they interact with the other classes. Cheers, - Ross

6 9

How to retrieve original string back from hashed string using string_hash()?
by Tiwari, Sumit [Tech] 07 Jun '10

07 Jun '10

Hi, I can hash a string using string_hash(), is there any function available to get back the string value back by supplying hash value? Thanks, Sumit

2 1

Re: [boost] Review of a safer memory management approach for
by Bartlett, Roscoe A 07 Jun '10

07 Jun '10

Fernando and Jeffrey, A few responses ... > ------------------------------ > > Message: 6 > Date: Fri, 04 Jun 2010 00:14:25 -0300 > From: Fernando Cacciola <fernando.cacciola(a)gmail.com> > To: boost(a)lists.boost.org > Subject: Re: [boost] Review of a safer memory management approach for > C++? > Message-ID: <hu9r1a$57q$1(a)dough.gmane.org> > Content-Type: text/plain; charset=UTF-8; format=flowed > > ... > > For instance, it's been so long since I last wrote a virtual function > that > recently, I made the silly mistake of forgetting that, in a derived > class, an > overrided function doesn't have to specify the virtual keyword ;) [Bartlett, Roscoe A] High warning levels (i.e. with GCC) catch most such mistakes that the language proper allows to go through. BTW, are you gravitating to stack-based programming because of the memory management problems or because of problems of change propagation of shared mutable objects? These are almost 100% unrelated issues in my opinion. Is the C++ boost community really ready to throw away OO programming? That just seems crazy to me. > ------------------------------ > > Message: 9 > Date: Thu, 03 Jun 2010 21:12:52 -0700 > From: "Jeffrey Lee Hellrung, Jr." <jhellrung(a)ucla.edu> > To: boost(a)lists.boost.org > Subject: Re: [boost] FW: Boost Digest, Vol 2929, Issue 4 > Message-ID: <4C087D44.1080909(a)ucla.edu> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > ... > > > As for sharing of objects, it is critical for many types of software > in the Computational Science and Engineering (CSE) domain. You have to > share large objects in order to cut down on runtime and storage > overhead. As one example, consider the Petra object model with the > Trilinos packages Epetra and Tpetra (http://trilinos.sandia.gov/) In > the Petra object model, you have objects with significant storage > overhead like Maps and Graphs that are shared between many different > objects like Vectors, MultiVectors, and Matrices. Yes, you could > create deep copies of Maps and Graphs but that would waste a large > amount of storage. The bottom line is that by sharing large objects > like Maps and Graphs instead of making large numbers of copies, we can > fit and solve much larger simulation problems on a computer than we > could otherwise. You could template and statically allocate everything > and you will never get around the fundamental need to share the data > contained in large objects > in > > these types of applications. > > > > Note, however, that Maps in Epetra/Tpetra are shared as const > immutable objects. That means that no-one can change a Map after it is > first created. Therefore, all of the problems with unexpected updates > goes away (as you mention above). However, the shared Maps must go > away when they are not needed anymore and that is what the Teuchos::RCP > class makes easy and robust. Under the covers, Tpetra uses > Tpetra::ArrayRCP for similar purposes and much more. The situation > with Graphs is different and issues of change propagation of shared > objects are still present in some cases. > > > > Come to think of it, most of the object sharing that used in the CSE > software that I use and write mostly just share const immutable objects > so problems of change propagation mostly goes away. However, there are > some important use cases where all of the clients are not holding RCPs > to const objects and the problem of change propagation remains. Again, > to save in significant runtime and storage overhead, we can't just > create deep copies of all of these objects. Templating and purely > stack-based programs are not going to solve that problem. > > > > The Teuchos MM approach, I believe, largely solves the memory > management problems with dynamic memory allocation and object sharing > while still yielding very high performance and mostly eliminating > undefined behavior associated with the incorrect usage of memory. The > most significant contribution of the Teuchos MM approach is the > presents of Teuchos::Ptr and Teuchos::ArrayView and how they interact > with the other classes. > > I don't see how shared object *ownership* is necessary in this > example... :/ I don't think Dave was arguing that holding large > objects > by reference, or even dynamically allocating objects (large or not), is > necessarily complex or "bad"... > > Or did I miss something? [Bartlett, Roscoe A] Have you read the key sections in the Teuchos MM approach in http://www.cs.sandia.gov/~rabartl/TeuchosMemoryManagementSAND.pdf or are you just responding to the list emails? You should *never* hold an object by reference as a class data-member (except perhaps in very well encapsulated very low-level code). That is what leads to undefined behavior with dangling references. Early versions of Epetra did just that and it blew up in people's faces. Consider writing a factory method such as: Epetra_CrsMatrix* createMyMatrix(....) { Epetra_Map rowMap(...); Epetra_Map colMap(...); ... Epetra_CrsGraph graph(rowMap, colMap); ... Epetra_CrsMatrix *matrix = new Epetra_CrsMatrix(graph); ... return matrix; } In early Versions of Epetra, maps and graphs were held and references and the above factory function returned an Epetra_CrsMatrix object with now deleted map and graph objects. Later, Epetra was refactored to make Epetra_Map and Epetra_Graph handle classes with internal reference-counted handle/body implementation objects. This fixed the problem but it create lots of new problems including that now copying graphs silently went from deep copy to shallow copy (which broke some code in some rare cases). The new Tpetra package has gone the down the road of making shared ownership explicit by exposing RCP-wrapped objects making factory functions similar to the above safe and obvious in behavior. It is working very well. Cheers, - Ross

8 8

Re: [boost] [endian] swap_in_place use case
by Dave Handley 07 Jun '10

07 Jun '10

Please don't drop attributions. Vicente Botet wrote: > Dave Handley wrote: >> Memory map a network endian file. Swap_in_place. Use. >> >> You definitely don't want a copy in this case since your file could easily be >> very big. Think of the case where your file is huge (say 10Gb), you really >> don't want to perform a copy and swap since that puts your memory need at 20Gb >> instead of 10Gb. > Yes this could be a use case. I don't use to manage so bigs files. If I had > to work with I will never copy the whole file. But I'm not sure that I will > use a swap_in_place of the whole file. This could take too much time. I would > try to split the task to do on the whole file in smaller parts. > > What will you do with this big file, that makes the swap in place the best choice? The file could be many things. It could be a day of market data for a given exchange. It could be image data or video data that I'm going to perform image analysis on (maybe run a filter over it, or something similar). The file doesn't even have to be that big. If I was memory mapping a 10MB file and needed to swap it, I wouldn't want to use 20MB instead of 10MB. Pretty much anything I want to do to that file that involves looking at most or all of the data you would be much better off using swap in place instead of any copying swap implementation. Examples of the sorts of things that you might want to do to large files include running filters or normalisers over image or video files. I have lots of programs that have multiple threads constantly memory mapping files that range in size from relatively small to hundreds of MB or low numbers of GB. Given that memory allocation is a key component of the run time of these programs, they would run significantly slower if I had to allocate double the amount of memory. Don't forget, if you need the whole file to be swapped, then the fastest way to do it will be a swap in place of the whole file. I will reiterate something I said in an earlier post. If boost accepts an endian library which does not provide an efficient swap in place, I will be unable to use it. The library will end up in the list of boost libraries which are too inefficient to use in performance sensitive production code. Dave Handley

5 6

The bug sprint has started!
by Marshall Clow 07 Jun '10

07 Jun '10

As of 0800 Pacific time today, we had 989 open trac tickets. Info about the bug sprint can be found at <https://svn.boost.org/trac/boost/wiki/BugSprintJune2009> By ticket type: Bugs 613 Feature Requests 238 Patches 95 Tasks 27 Support Requests 15 Library Submissions 1 By milestone: To Be Determined 187 Boost 1.43.0 180 Boost 1.42.0 132 Boost 1.40.0 85 Boost 1.36.0 82 Boost 1.41.0 81 Boost 1.39.0 67 <none> 61 Boost 1.38.0 45 Boost 1.37.0 37 Boost 1.44.0 20 Website 1.X 10 Boost.Jam 4.0.0 1 Top 10 libraries: thread 76 date_time 55 Python 48 test 43 interprocess 40 iostreams 37 filesystem 36 mpl 36 quickbook 33 asio 31 Top 10 owners: anthonyw 75 djowel 71 dave 58 igaztanaga 58 az_sw_dude 53 vladimir_prus 49 bemandawes 46 rogeeff 43 agurtovoy 36 turkanis 36 Useful reports: https://svn.boost.org/trac/boost/report/1 -- full list of tickets https://svn.boost.org/trac/boost/report/18 -- ticket counts by owner https://svn.boost.org/trac/boost/report/19 -- ticket counts by component https://svn.boost.org/trac/boost/report/20 -- ticket counts by ticket type https://svn.boost.org/trac/boost/report/21 -- ticket counts by milestone

3 3

Re: [boost] Review of a safer memory management approach for C++?
by Bartlett, Roscoe A 07 Jun '10

07 Jun '10

Hello, This is the last email that I will write and respond to on this tread of emails about runtime vs. compile-time polymorphism (we have beet this dead horse about enough I think). I will summarize my arguments and the baulk responses from people. My perspective comes from having developed and maintained a templated package in Trilinos called Thyra that uses only the most basic type of templating; templating runtime abstract interfaces on the single Scalar type (i.e. double, float, complex<float>, complex<double>, etc). This is the most trivial and basic type of templating that you can possibly imagine (and was the type of templating that templates in C++ was supposed to support in the first place). Templating all of this code was not my first choice but in retrospect it was the right thing to do since you would have to duplicate all of the code (source and binary) multiple times (once for each scalar type) in order to support this variety of scalar types in C++ in a single program. However, this has not been easy by any stretch of the imagination. Now Trilinos is redoing is core linear algebra and linear solvers with the advent of the more heavily templated Tpetra package and will have similar issues. Up until recently, I was just having to compile and link Thyra and client code using the implicit template instantiation mechanism on the *wide* variety of compilers that we are contractually obligated to support including several different versions of GCC, Intel, Sun, PGI, IBM, Pathscale, and Visual C++ on a whole bunch of platforms (Linux, various flavors of Unix, Windows, specialty HPC systems, etc.). In many ways, I think that I have been the unfortunate first adopter of heavily templated code on many of these platforms. Eventually, I figured out how to portably and maintainability manage explicit template instantiation (see GCG 28 in http://www.cs.sandia.gov/~rabartl/ThyraCodingGuideLines.pdf) which has allowed me to port this templated code to several platforms where it would otherwise just segfault or crash the compilers (e.g. Sun and PGI). Going to carefully orchestrated explicit template instantiation has allowed me to recover a lot of the positive properties of non-templated code (i.e. fast recompile times, being able to compile just a single source files to work out compile problems, etc.). However, even with this most trivial type of templating (a single template argument), explicit instantiation is not automatic to support and significantly complicates the design and maintenance of the software. For new more heavily templated packages like Tpetra and the next generation of template software based on Tpetra in Trilinos that is being written, it is even more difficult and more complex to support explicit instantiation. If have fought the good fight for the use templated code in this non-ideal HPC environment for many years and I have earned to right to express my experiences and wisdom from the trenches on this topic. In return, the majority of the comments have basically been: a) Common response from list members: If have had trouble with templated code, it is because I don't know how to design or write software (just read some of the comments below and in some of the other email responses). My response: You can't get more simple templated code than a single Scalar template argument that is used in Thyra. I can't imagine a more basic use of templating and I have no clue how to design this "better". The only way to make this portable is to use explicit template instantiation which is not completely trivial to support, even for the most simplistic type of templating (i.e. just one Scalar template argument). b) Common response from list members: I should not bother supporting troublesome compilers like Sun, IBM, and PGI. My response: As I have explained many times, if our code can't support *all* of these platforms, then our customers can't use any of this code and we need to find another job. I don't know how much more direct I can state this. I have fought many battles against our customers who demand to use bad C++ compilers but we are going to have one or two bad compilers that we need to support for many years to come that throws a monkey wrench into the otherwise "perfect design" of the software we would like to write. You should see the emails that I get from people about "being in an ivory tower" when I complain about having to support non-ANSI C++ compilers. If some of my responses have been testy, it is because I am a little warn out with the above two general responses to problems that I have had with heavily templated code. Please don't just dismiss my opinions based on hard fought experience by assuming that I have no idea what I am doing. This entire thread of many emails and responses was sparked by my original request to review the approach described in: http://www.cs.sandia.gov/~rabartl/TeuchosMemoryManagementSAND.pdf which really has nothing to do with runtime vs. compile-time polymorphism. There has been a lot of responses but has anyone taken the time to even read at least the reduced table of contents described in the Preface? I know this is a long document at over 150 pages in the main text but the reduced table of contents is only something like 45 pages or so. A few selected responses to some other emails below ... > 1. Re: Review of a safer memory management approach for C++? > (Mathias Gaunard) > 3. Re: FW: Boost Digest, Vol 2933, Issue 1 (joel falcou) > ---------------------------------------------------------------------- > > Message: 1 > Date: Sat, 05 Jun 2010 17:59:24 +0100 > From: Mathias Gaunard <mathias.gaunard(a)ens-lyon.org> > To: boost(a)lists.boost.org > Subject: Re: [boost] Review of a safer memory management approach for > C++? > Message-ID: <hudvpc$76f$1(a)dough.gmane.org> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > ... > > > 2) Current compilers may not even be able to compile large amounts of > template code at all. There are compilers that we have to support > (e.g. Sun and IBM) that will segfault or return "internal compiler > error" when you give them too much template code. These same compilers > have no problem with virtual functions with standard OO. This is > reality. > > It's up to you whether you want to support those compilers or not. > I myself see little compelling reason to support the Sun compiler, > almost likewise for IBM, at least for new C++ projects. > > Balance your support for legacy compilers with your willingness to have > nice code. [Bartlett, Roscoe A] As explained above, I fight the battle all the time to avoid having to support such bad non-ANSI compilers but we will have to live with these problems for many years to come. If we can't support *all* of these platforms, then they can't use *any* of the code that does not port to all of these platforms, period. This is not a joke. I suspect that Trilinos and other HPC projects are not the only software projects that have to live this reality. > > 3) Templated programs can result in massive object-code bloat over > equivalent runtime OO programs. If you are really sloppy with > templates and use it for everything, you can easily double, triple, or > more (10x) the size of the object code and executables (see Item 44 in > "Effective C++, 3ed edition"). > > They can, but they don't have to. > If your template function is not a good inlining candidate, is likely > to > get instantiated with many different types and and if making it > runtime-generic has sufficiently low costs, then you should consider > doing it. [Bartlett, Roscoe A] 99% of Thyra-based code should not ever be inlined (see description of Thyra above or just go look at the Thyra Website). > Note, however, that the size costs are usually grossly exaggerated. > I've > seen people use advanced template C++ code on microcontrollers (the > kind > with a few kilobytes of memory) just fine. [Bartlett, Roscoe A] If you templated the algorithms instead of using Thyra interfaces, you would have new and different object code for every concrete set of operator/vector subclasses. Just in Trilinos this would result in at least a 3x increase in the size of the object code. There is no way around this. > > As long as people keep suggesting that we template all of our codes > for every little reason > > No one is suggesting this. > Make your code only as generic as your algorithms intrinsically are, no > more. > > Algorithms are what lay at the center of generic programming, not > objects. This is quite a different view from OOP. [Bartlett, Roscoe A] Using Thyra as the example, if you throw away the abstract interfaces, then 99% of your algorithmic code would be templated in order to achieve (compile-time) polymorphic behavior. This is a very simple issue to analyze. > > Thinking that we are going to drop runtime polymorphism and use > exclusively compile-time polymorphism for everything is just fantasy > land. > > Compile-time polymorphism is better, and you can build runtime > polymorphism on top of it with type erasure. > > So why avoid compile-time polymorphism when you can do it? [Bartlett, Roscoe A] The only way to manage large amounts of templated code is to explicitly instantiate it which is not trivial to code and maintain (again, see above). > ------------------------------ > > Message: 3 > Date: Sat, 05 Jun 2010 19:08:55 +0200 > From: joel falcou <joel.falcou(a)lri.fr> > To: boost(a)lists.boost.org > Subject: Re: [boost] FW: Boost Digest, Vol 2933, Issue 1 > Message-ID: <4C0A84A7.7060002(a)lri.fr> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > ... > > And I can tell you that, as a consultant and now as an assistant > professor involved in variosu HPC project, the quality of code I saw in > laboratory is usually sub-par. [Bartlett, Roscoe A] What is this supposed to mean? > > The size of object code does have some impact, especially some HPC > machines. Again, please teach us all how to do this because we > apparently don't know what we are doing. > > > You can also learn to reply without actually barking at people. It > helps > having a sane conversation. [Bartlett, Roscoe A] As described at the top, when the common response to problems is that "you don't know how to design software" one has almost no choice but to respond in such a way. Again, the templating in Thyra could not be more simple. > > The compilers segfault or return internal compiler errors or take > hours to compile if they do finish at all in some cases. See above. > Reality!!!!!!! > > > Reality of sucky compilers from 2 century ago. It's lkike blaming > people > using new cars with biofuel whiel comapring them to Ford T. [Bartlett, Roscoe A] As described at the very top, I fight the battle against bad C++ compilers constantly but we are going to have to live with some bad compilers for many years to come. What do you expect us to do? Should I just quit my job as a protest against bad C++ compilers? > > My argument is that current accepted ways of writing OO code in C++ > make it too hard to write correct well designed programs and an > affordable cost. My argument is that the Teuchos MM approach described > in: > > > > http://www.cs.sandia.gov/~rabartl/TeuchosMemoryManagementSAND.pdf > > > > together with static analysis tools to help enforce the idioms (and > keep raw pointers encapsulated) can largely eliminate undefined > behavior in C++ programs due to the usage of memory and also yield > programs that are more self-documenting at the same time. > > > > > Well, you better be writing JAVA or OCaml ... [Bartlett, Roscoe A] Not true. As described in Section 6.1 in: http://www.cs.sandia.gov/~rabartl/TeuchosMemoryManagementSAND.pdf the idioms using the Teuchos MM classes can yield better self documenting code than you can get with languages like Java and Python. Did you bother to read the document? Yes I know it is very long but I think there is some worthwhile material contained in it that should be given a far shake and not just dismissed by uninformed email responses. (By uninformed, I mean you likely did not ready the document, not that you are fundamentally uninformed). - Ross

2 1

[thread] Customizing barrier for improved performance
by Belcourt, Kenneth 06 Jun '10

06 Jun '10

Hi, I notice that the thread barrier class is fairly large (128 bytes on Darwin with Intel 11.1). private: mutex m_mutex; condition_variable m_cond; unsigned int m_threshold; unsigned int m_count; unsigned int m_generation; and sort of slow for my application (parallel iterative solvers of sparse linear systems). Many iterative algorithms have both serial and parallel sections during a single iteration and, for larger algorithms, this can result in numerous (order 10 or so) rendezvous points during each iteration. During cursory testing I've found that a barrier implemented with atomics is a bit faster than a mutex based barrier (though I recognize that an atomic spin-based implementation can potentially hang if running on a single Intel core with hyper- threading enabled). I've attached a simple atomic based implementation built on Intel tbb atomic though it's easily convertible to boost.atomic when the time comes. This implementation just ping-pongs a counter alternating between incrementing and decrementing the counter each time it's called. Does anyone know if there's plans to extend barrier so that a user could select a different implementation (like an atomic based one)? For some applications this could be a very useful extension. Thanks. -- Noel

1 0