- Boost - lists.boost.org

Do really property_tree and program_options overlap?
by Marcin Kalicinski 23 Apr '06

23 Apr '06

Hi Everybody, I see there is some confusion about possible overlap of program_options and property_tree libraries. First of all, my opinion is there is very little overlap. The biggest difference between the libraries is that property_tree is hierarchical and program_options is linear - i.e. options are accessible through a map-like interface. Structure of file formats, like XML, JSON etc. is inherently hierarchical. The presence of hierarchy is their greatest strength, and most important feature. Large amount of information is conveyed by structure of the tree alone, not only by values it contains. The same value in two different branches of the tree can have completely different meaning. Contrary to that, in program_options library there is only at best a notion of "positional options", which is completely different from hierarchy. Briefly, PO library operates on a flat strings of options, property_tree is a DOM. Think of property_tree like a replacement for Microsoft XML parser, or JSON API, or Windows INI API, or registry API. Do you imagine using program_options to manipulate XML trees, like the ones you get from XML parser? It is not suited for that, its problem domain lies in completely different place, namely parsing a linear string of (command-line) options, possibly supplied as a config file or files. In addition to that, PO library does not support writing the structures back to config files. It is not a flaw with the library, it just does not need to do it, because options for programs are meant to be "one shot", or "read and forget", like command line arguments. There's not need to store them back where they came from. On the other hand, PO library supports many things that do not belong to property_tree. These include, for example, options descriptions and notify mechanisms. Taking all that into account I do not see why anybody insists there might exist 1:1 correspondence between the libraries. I agree both can parse command lines, but that's about it. Other functionality is very different. One has to distinguish between reading command-line options (or other simple configuration) and reading structured DOM data. For example, take a program like visual XML editor. It could use both PO and property_tree to read its startup config (keyboard shortcuts, font etc.). But only property_tree would be suitable to handle XML files to be parsed, displayed, edited and saved by the editor. It would be rather ridiculous to use program_options to do that, although not impossible, if one tried really hard. Best regards, Marcin

6 12

Re: [boost] [Boost Review] Property Tree Library
by Tom Brinkman 23 Apr '06

23 Apr '06

Marcin wrote > I think I understand your point, although you did not state it directly. > (Please correct me if I'm wrong) you believe basic_ptree class has some > value, and meets boost quality standard. On the other hand you think that > parsers that accompany it do not. I dont have a strong opinion one way another about the parsers meeting or not meeting boost standards. I just want to make shur that they get reviewed and let the boost members voice there opinions about them. > However, I'm also sure you are aware of the fact that without _any_ parsers > the library is quite useless. It might still have some obscure uses, but at > least 90% of it is gone. I'm not aware of this. Please explain why the "property container" is useless without a parser. This is the critcal reason why I have issues with this library in its present form. Why is "property container" different from other containers such as vector or map in this regard. > Therefore I can only agree to your proposition if > at least _one_ parser is included with the library. Which one is another > issue, but I believe it should be XML. That may be accaptable. However, its the XML parser that is my main concern. That would be the one that I would save for a later review. > Also please note that while at the moment some parsers may look ugly, it is > the implementation that is ugly, not the interface. And implementation can > be safely improved while the library is already in boost. Noted. Tom Brinkman Boost Review Wizard

2 1

[Property Tree] Some initial Review Thoughts / Questions
by Jeff Garland 23 Apr '06

23 Apr '06

This isn't my full review, just a few issues I see. I think this library has great potential and value, but I'm going to stick to the issues for now. I posted most of this just before the review, but got no response: http://lists.boost.org/Archives/boost/2006/04/103305.php I see 3 serious issues at this point: 1) Find is under documented / doesn't work for me ptree pt; read_ini("test.ini", pt); ptree::const_iterator ci = pt.find("value2"); if (ci != pt.end()) { std::cout << ci->second.data() << std::endl; } //test.ini [User Data] ; This is a comment value1 = v1 value2 = v2 date = 2006-Apr-18 I tried various keys in the find including: ptree::const_iterator ci = pt.find("User Data.value2"); All returned null iterator. So there are two problems. From the docs it wasn't clear what syntax to use with find and it doesn't appear to work. 2) Iterators are underspecified It was totally unclear to me from the docs what the interface to iterators returned from the interface. I finally found a test or example and copied. std::cout << ci->second.data() << std::endl; 3) Write operations are not 'whitespace' and comment preserving To me this is a critical point not fulfilled by the library. Most real-world configuration files have comments. If the application needs to read/write the files it needs to maintain the comments as human written. In my experiment if I start with this: //test.ini [User Data] ; This is a comment value1 = v1 value2 = v2 date = 2006-Apr-18 After I write it back I get: [User Data] value1=v1 value2=v2 date=2006-Apr-18 I realize that this requirement might increase the resource usage of property_tree, so I'm fine with this being an optionally enabled feature. Jeff

4 5

Re: [boost] [Boost Review] Property Tree Library
by Tom Brinkman 23 Apr '06

23 Apr '06

*>What value does the library have, if there are no parsers to get >a property tree from or to write it back to a file? > >* > I fully agree with this point. Removing the parsers from the review > makes the library essentially useless. They are an important part of the > functionality the library provides. A simple string tree is not enough > functionality to be relevant; if only a tree was to be submitted, I'd > expect it to be a fully generic tree. Please, RE-READ what I said. I want this review broken up into two parts. I said nothing about the value of this library being useless without the parsers. Thorsten, please contact me off-list about this.

3 2

Re: [boost] [Property_tree review] Summary of enhancements proposed so far
by JOAQUIN LOPEZ MU?Z 22 Apr '06

22 Apr '06

----- Mensaje original ----- De: Marcin Kalicinski <kalita(a)poczta.onet.pl> Fecha: Sábado, Abril 22, 2006 2:48 pm Asunto: [boost] [Property_tree review] Summary of enhancements proposed so far > I created a brief summary of enhancements/changes to property_tree > which > were proposed so far. > > 1. Make path a class in its own right and overload / (division) > operator to > concatenate path objects efficiently. This removes need for ugly > default > separator, and half of the get/put overloads. It also boosts > performance (no > more string operations on path), and IMHO will generally improve > interface > of the library. Most importantly, it largely does maintain > compatibility > with existing interface. Although a proper path class seems so much neater, I think you should still keep some API to specify a path as a single string, maybe something like: boost::property_path::path p("Dante/Divina Commedia/Inferno"); Why? Because this way you retain the ability to store a path as a program resource outside the code. Joaquín M López Muñoz Telefónica, Investigación y Desarrollo

4 3

[Property_tree review] Summary of enhancements proposed so far
by Marcin Kalicinski 22 Apr '06

22 Apr '06

I created a brief summary of enhancements/changes to property_tree which were proposed so far. 1. Make path a class in its own right and overload / (division) operator to concatenate path objects efficiently. This removes need for ugly default separator, and half of the get/put overloads. It also boosts performance (no more string operations on path), and IMHO will generally improve interface of the library. Most importantly, it largely does maintain compatibility with existing interface. 2. Split Traits into smaller policy classes. For example KeyPolicy and DataPolicy. Indeed, these two are mostly orthogonal, and it should be possible to replace one without touching the other. 3. Allow use of other data types as keys/paths (for example a vector of ints). Initial support is in the library, but key is still required to support string-like interface. 4. Add possibility to disable indexed lookup, if not required 5. Add possibility to specify names of "special" keys in parsers (<xmlattr>, <xmlcomment> etc.). Add whitespace-handling flags to XML reader. 6. Add "set" function. "set" will work in a similar fashion to put, except that it will always replace existing key if it exists. Extra boolean parameter to put functions will be gone. Alternative approach is to replace bool parameter to put with enum to improve readability. 7. Add "del" function. Del will erase existing keys from the tree, while presenting get/put/set style interface. This is clearly missing, although not immediately obvious. 8. Add support for array-style indexing in paths. This will probably work well with #1. 9. Add support for serialization of ptree objects 10. Add possibility to iterate over the whole tree in one go 11. Use multi_index internally for indexing 12. Make get/put functions freestanding 13. Add more docs on customizing key/data types. 14. Improve docs grammar/style in places. Break large index into smaller files. Additionally there were proposals to use runtime polymorphism to distinguish between leaf and branch nodes. I think this idea strays so far from current implementation/interface of the library, that it is not feasible to do. I hope this will serve as a basis for continued discussion. Best regards, Marcin

2 1

Serialization and Editing Archives
by Robert Ramey 22 Apr '06

22 Apr '06

I've been looking at the posts regarding the Property Tree Library. I haven't had the time to give it a proper review. Fortunately it seems that there are a good number of people willing to help with this. I would like to make some observations regarding the serialization library and the creating of "editable" archives. This seems to me the central issue being discussed. The fundemental notion that the serialization library is meant to address is the deconstruction of an arbitary set of C++ data structures into a series of bytes in such a way that the original set can be restored at another time and place. The requirements for such a library were a) completeness - basically no constraints on the C++ data structures to be saved/restored. b) factoring out the actually storage of the bytes in such a way that this could be changed by the user to suit his application. c) minimal effort - the the library has to leverage on the structure of the data as described by the C++ code itself rather than than requiring some extra specification as to how all the data is organized and related. This goal has been in large part achieved with the help of many people on this list, The set of requirements was demanding, and required lots of code to implement. Much of that code had to relie on more demanding techniques such as template metaprogramming. Also, in some cases, extra information had to be be appended to "reflect" information necessary for the implementation. BOOST_CLASS_EXPORT is an example of this. In addition a number of user customizable attributes had to be defined to permit the library to be applied as widely as was reasonable to expect. I know not everyone is totally happy with the library but all in all I'm satsified that a good balance has been struck. The goal and requirements dictate that the resultant archive (however it is stored is dependent on the C++ data structure defined at compile time and invoked at runtime. In general, one cannot expect that any changes in an archive will possible without changing the C++ data structures and code to which the changes correspond. Note this is an entirely reverse situation to the most common one exemplified by doing something like parsing an XML file. In this case one will load the data into some dynamic data structure which will reflect the structure of the data being loaded. At that point some code will have to traverse the structure to extract or set the data in memory. Note that there is no requirement that the C++ data structures map in any special way to the file. Also the file doesn't have to include as much "detail" because it doesn't have to reconstruct an arbitray set of C++ data structures. And of course we see and will continue to see a proliferation of file formats and programming modules to support them. Some attempts have been made to map serialization to some formats which permit arbitray data - XDR and a couple of others. I don't think they've been successful because part of the file format is really held in the C++ code which saves and loads the archives. The requirement to be able to arbitrarily edit an archive is clearly not possible. But then ... it seems tantalizingly close - I meant hell its XML I can read it!. So could anything be done? I haven't spent too much time on this but I can speculate some. Suppose one wanted to create and editable XML format. First of all on will have to accept the fact that not all editing is going to be possible. One won't be able to make a new class which the C++ code doesn't implement. I might be possible to do something like the following; a) parse the whole XML file with a standard parser b) as the serialization algorithm visits each data structure, ratther than just presuming that all the data is there in sequence, it would check to see if the tag has been loaded.. This would permit the following types of editing: i) deletions ii) changing data A little more work would permit expanding / contracting collections. I really don't know how much work this would end up being but no one has had sufficient interest to undertake it - or perhaps they have but it hit a dead end. To summarize I see two almost irreconciliable paths a) Archive formats reflect the set of C++ data structures which generate them b) A format for arbitrary data (e.g. XML) is defined and a general- not application specific - data strucure (e.g. tree) is defined. Then the application walks the tree in an application dependent manner. Basically only one thing can be boss - either the C++ data strucures in the app or the file format. one can drive the other but they can't be varied independently. It seems to me that the property tree library is driven by the need to be able to edit one's "serialized" data out side the main application. My personal view is that if one feels that he needs to do this - he should be rethinking his application. A huge portion of applications are really variations on just this theme: Load some data, edit it, save data. And its not just word processors or things like that. Machines startup, run, and shutdown. a customer recored is accessed, updated, and saved. etc. So I think many more applications can be considered in terms of serialization than in fact are. The applicatons edit data. There can also be the case where different applications edit the same data. I might make an application which loads configuration, runs a machine and save the data from the run. Someone else needs and application to browse and/or annotete the the data. One could say - oh then we need the data in some application independent manner such as XML. Well that's one way - but then we have to recode all over again to get and save the data. Or we can use something like a text editor but then we are casting aside all the facility of using a computer in the first place. Finally we can make a new application which used the original C++ serialization code and data structures - which hopefully have been compiled into a library. Now we just use our special editor for that "other" applicaion. To me - definition of a file format independent of the C++ code which is going to use has wasted billions of man years of human brain time just trying to keep things in sync. So, I see the whole XML approach as a misguided attempt to separate the fundementally inseparable - data and code - they are one now. I predict that XML will be made continually more elaborate and spawn ever more variations in vane attempts to reconcile the unreconcilable. Remember you heard it here first - XML and is progeny is destined for the ash heap of history! (hmmm - like COBOL?) For someone who has nothing else to do there are interesting things to look at here. What would be very interesting would be a xml_archive which created in parallel an (XML?)_schema which would be used by a generic editor to edit the xml (and maybe other) data in the archive within the bounds of what is permitted. I see this as quite doable. Another idea would be a generic editor which could be compiled from the serialization code which would exploit the "reflected" data about the C++ and perhaps define a little bit more. For example I could imagine a template for each datatype which would invoke a functino to edit it. On a GUI, just compile the generic editor with your own serializtion code. and BAM you've got an application specific editor which runs at full runtime speed. Just a couple of misceleaneous observations a) It seems there is a perception that the serialization library is large, cumbersome and inefficient. I believe it can be used inefficiently (like all libraries) and it can be made faster (and incremental improvemetns are being made). But I don't think this is true in general. When I compile with release mode, make sure I haven't included in the debug symbols, and use the DLL (which isn't all that big), a program which saves and loads 4 different types of archives with several classes only takes 30 KB. So if you're not getting similar results, take a closer look. b) much has been made of things like - you can't turn off tracking etc and the difficulty of modifying the library. I believe that most of these difficulties comeabout by tyring to implemnent some external requirement - which I've called about trying to reconcile the irreconciable. If one turns off tracking- then how is one going to recoveer pointers. By default, someting track only if its used as a pointer somewhere. And why would one have pointers in something like configuration parameters. And if tracking is turned off and somethng is duplicated - then what if one edits only one of the copies. etc. Its not so much that I want to dicuss tracking- its more that I think these kinds of questions reveal that things originally thought to be almost identical e.g and XML archive and general XML file are more different than they first appear. So when this occurs its time to step back and look at the big picture (BTW I believe it would be easy to suppres the object_id etc in he xml archive just be overridding with stubs the serialziations for these types - but then how would one read the archve?) Anyway - I'm sorry if this is a little rambling and off topic Robert Ramey

1 0

Re: [boost] Do really property_tree and program_options overlap?
by Matt Doyle 21 Apr '06

21 Apr '06

> -----Original Message----- > From: boost-bounces(a)lists.boost.org > [mailto:boost-bounces@lists.boost.org] On Behalf Of Marcin Kalicinski > the lot. You can use BOOST_FOREACH on it. You can use > algorithms on it (just keep in mind it is not sorted by key > unless you do that explicitly). Are you talking about basic_ptree::sort? Or pre-sorting the keys before populating the tree? Scanned by McAfee GroupShield {X3BTB534}

2 1

[Property tree review] Error in docs
by Marcin Kalicinski 21 Apr '06

21 Apr '06

I just noticed there is a serious error in docs, it states that parsers are in namespaces of their own, (e.g. read_xml is in namespace boost::property_tree::xml_parser). This is correct, but they are also all imported into boost::property_tree namespace by means of using declarations. And they should be used from there. Their presence in other namespaces should be considered an implementation detail, and obviously not documented. I will fix that after review completes. In the meantime please bear in mind that there's not so much namespace bloat as the docs purport ;-). One reason that parser functions _must_ be in property_tree namespace is Koenig lookup, e.g.: boost::property_tree::ptree pt; read_xml("x.xml", pt); // Koenig lookup here Lookup above obviously relies on read_xml being in the same namespace as ptree. Marcin

3 3

Re: [boost] [Boost Review] Property Tree Library
by Matt Doyle 21 Apr '06

21 Apr '06

> [mailto:boost-bounces@lists.boost.org] On Behalf Of Thorsten Ottosen Snip> > > Performance is a concern given the recursive nature of the > tree. I want write a test case parsing Usenet's group list. > By it's nature it demands storage in a hierarchical tree > where every node is both a leaf and a branch (like ptree :) , > it's a big list, and insertion / retrieval is mostly random. > Hopefully I'll have time to do this before the review is over. > > Snip> > > Let me just say (as a review maanger) that such a test would > be immensely useful (an an obvious candidate for inclusion > with the library if it is accepted). > > I know it is an easy task, though. Well, I finally have some performance results and, at to me at least, it looks pretty good. I used a partial snap shot of the Usenet groups list with 34767 records in it. I choose that particular file because I had previously used it to compare performance between MFC's TreeView and a handrolled tree using Berkley DB_TREE. TreeView parsed it in ~1200 seconds, Berkley at ~40 seconds, and now Ptree came in at 7.5 seconds. To be fair, I didn't re-run the first two tests today, I just used my results from previous tests so that may have been a slower computer (~two years ago). Anyway, of the 7.5 seconds 4.3 of it was I/O time so the library time to build & populate the tree was 3.2 seconds, or 92uS per node. Iterating the tree and writing it back out took 2.187 seconds, the average seek time to search for a record was 16uS and the average time to search and modify the same record was 125us. SO all that being said, I want to re-affirm my vote that Ptree should be accepted. One thing aside though, I want to harp on the documentation again. While the library was easy to use there were a few nuances that I had to research along the way. I think that at a minimum we need to have some concrete examples, starting at main(), showing the usage of each of the parsers. Attached is the test dataset and the test program if anyone else wants to independently retest it. Regards, Matt Scanned by McAfee GroupShield {X3BTB534}

1 0