- Boost - lists.boost.org

Formal review: serialization
by Jeff Flinn 26 Apr '04

26 Apr '04

I was not involved in the original review. > Do you think the library should be accepted as a Boost library? Yes. > What is your evaluation of the design? The design, from a usage standpoint is excellent. Coming from the MFC world, boost::serialization far surpasses MFC's in usability and flexibility. Particularly notable are: no need for a common base class, a single serialize method used both for input/output. The only problem I see is that programming errors in some cases can be difficult track down, but no more so than with any template intensive library. While the current methods for Free Functions and Splitting save/load methods are certainly understandable and work. It would be nice to be able (partial)specialize the "serialize" function base on its IO type. It would also be nice to utilize enable_if to generate different code when the serialize function is instantiated for input/output archives. This would allow compile time constructs similar to mfc's runtime Ar.IsLoad(). > What is your evaluation of the implementation? I've been using the serialization library since version 13. All previous version bugs/issues have been cleared up with version 18. I've been able to remove all workarounds from my code, in particular dealing with support of multiple archive types in a single translation unit including binary, text and xml. Performance is more than adequate for my purposes. My application uses binary serialization to implement drag/drop. On a 3 year old 2GHz pc, a pair of save/loads of a ~2MB including graphical updates takes less than 2 seconds. I have only briefly viewed library source code when finding workarounds in previous versions. Others here are much more qualified to render a verdict in this regard. > What is your evaluation of the documentation? The documentation allowed me to add serialization to my application without needing to look at header/implementation files. Though, I'm sure I carry some context from having worked with MFC serialization. One misleading aspect IMO is with the Non-Intrusive Version example. The example implies that you would need access to the data members. I think an example of what to do when you don't have access to the data members is needed. For example, how would one serialize a boost::filesystem::path object. namespace boost { namespace serialization { template<class Archive> void save( Archive & ar, const boost::filesystem::path & t, unsigned int version ) { std::string lPathString( t.native_file_string() ); ar << make_nvp( "PathString", lPathString ); } template<class Archive> void load( Archive & ar, boost::filesystem::path & t, unsigned int version ) { std::string lPathString; ar >> make_nvp( "PathString", lPathString ); t = boost::filesystem::path( lPathString, boost::filesystem::native ); } } } BOOST_SERIALIZATION_SPLIT_FREE( boost::filesystem::path ) I'm sure this could be improved upon, but without it, there is a danger that many will think - "can't be used for my application". Seeing an examples like this makes it a little easier to remember that the template functions need to be defined in the boost::serialization namespace. > What is your evaluation of the potential usefulness of the library? I've always considered serialization to be a major element missing from C++. Nearly every application uses some method of custom serialization today. I've saved 100's of hours already in using boost::serialization rather than spinning custom file formats or implementing connections to DBMS systems to store/retrieve data. > Did you try to use the library? With what compiler? Did you have any problems? Yes, with VC71. As stated above, all problems with versions previous to version 18 have been corrected. One, recent problem is an access violation when loading a saved shared_ptr(NULL). > How much effort did you put into your evaluation? A glance? A quick reading? In-depth study? Applied in an actual development project over the last several months. > Are you knowledgeable about the problem domain? I've some experience with MFC serialization. Thanks for a great library! ----------------- Jeff Flinn Applied Dynamics, International

2 3

[Boost-bugs] [ boost-Support Requests-942349 ] Problem compiling :/
by SourceForge.net 26 Apr '04

26 Apr '04

Support Requests item #942349, was opened at 2004-04-26 07:19 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=207586&aid=942349&group_i… Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: Nobody/Anonymous (nobody) Assigned to: Nobody/Anonymous (nobody) Summary: Problem compiling :/ Initial Comment: Hi :) I've some problems compiling code that use \boost\iterator\transform_iterator.hpp (under VC++ 7 - VS2K2) : transform_iterator.hpp(45) : error C2825 : 'UnaryFunction::result_type': cannot form a qualified name transform_iterator.hpp(45) : error C2039: 'result_type' : is not a member of 'operator``global namespace''' transform_iterator.hpp(45) : fatal error C1507 : previous user errors and subsequent error recovery halt further compilation Thanks in advance :) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=207586&aid=942349&group_i… ------------------------------------------------------- This SF.net email is sponsored by: The Robotic Monkeys at ThinkGeek For a limited time only, get FREE Ground shipping on all orders of $35 or more. Hurry up and shop folks, this offer expires April 30th! http://www.thinkgeek.com/freeshipping/?cpg=12297 _______________________________________________ Boost-bugs mailing list Boost-bugs(a)lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/boost-bugs

1 0

Re:[boost] Re:Re:Formal review: serialization
by Robert Ramey 26 Apr '04

26 Apr '04

Vladimir Prus wrote: > > Robert Ramey wrote: > > I guess I have two questions: > 1. Won't serialization fail in some way if I just seek the stream to the > position found in index and try reading. > 2. For random access I need to make sure that all saved objects have > export > key. How do I do that? Not necessary out-of-the box but where I can plug > the check? > I believe the original question referred the ability to "browse" the archive for purposes of debugging etc. My view would be that the best way to do this would be to create and xml_archve. Your use case suggested a 30 data would probably would end up as a 90MB XML datafile. The reason for thinking in XML would be that I would hope that here exists "XML Browser for large files" which would be suitable for this case. If such a thing doesn't exist - I would prefer spending time making this general purpose tool - (and maybe a commercial product) to making a custom archive of narrow purpose. Random access into an archive would require some thought. First, it's not clear how it would be used. In general archives are nested structures, one could de-serialize an inner piece - but to where? The original data structure was embedded in something else which is not now there. So this can really only consider in the context of a specific application. Here are a couple of plausible scenarios. a) A general purpose archive browser - this could browse the archive in a random manner but wouldn't actually de-serialize the data. I don't see any real problem here. One would need to create an index either as a side effect of archive create or with a second pass over the final archive. b) using serialization for a data-state logging. Log(Archive ar, statedata){ // save seek point to index // append to archive ar << statedata; } Recover(Archive ar, statedata &, seekpoint) // set stream seekpoint ar >> statedata } I could envision something like this being made to work. So I would say that generally, serialization would be expected to be a serial operation. (surprise!). On the other hand, in certain special situations it might be possible/convenient to provide for some random access but I would expect that to be application specific. > >> The reason why I think the other is important, it that's it's actually > >> saving/loading support for plain C++ array -- which is rather basic > >> thing. > >> > > > > Hmmm - the library already implements serialization of plane C++ arrays > by > > serializing each element. > > For *fixed-size* arrays. But not for dynamically allocated arrays. BTW, it > seems we need two wrappers for completeness: > one for dynamic arrays with element-wise save I would envision one using ar << element_count For(i = 0; I < element_count; i++) ar << element[i] I'm not sure that its worth adding and documenting such an obvious and trivial thing as a separate wrapper. > and another for dynamic arrays with binary save. I see the binary object as filling that role. Ar << make_binary_object(dynamic_array_address, element_size * element_count); I once did propose a manual section name "Serialization Wrappers" which would have a couple of examples and highlight the fact that nvp and binary_object are instances of this concept. The idea was received unenthusiastically at the time but now I'm becoming convinced It's more effective to explain and illustrate the concept rather than try to anticipate all the wrappers that users might need. Actually, I think having such a section would have avoided the confusion surrounding the intended usage and purpose of binary_object. > I agree. I actually have a *crazy* but cute idea that one can use file > offset for object id. How object ids are assigned and can I customize that > process? That would keep overhead at absolute minimum. Object_id are assigned sequentially starting with 0. They are only used for classes which require tracking (e.g. when instances are serialized through pointers) they are used as indices into a vector so using these indices keeps overhead to a minimum. There are cases when an object id is assigned but not written to the archive I don't see these as having much utility outside of serialization. > Ah, I've missed that. Do I need to provide both 'type' and 'value'? Can't > serialization library work with just one? Could be. I just provided both so that I could interoperate with mpl without having to think about each specific case. > >> It also necessary to provide overload for char*. Does not count as > >> primitive > >> type? > > > > Default implementation for char * will work as it does for other > pointers. > > Eh... I though that serialization of pointers to builtin types is just not > allowed. > Default implementation level for pointers to primitives is set to not-serializable. This is resetable as it is for any other type. I doubt that on would want to do it for something like int or double as I would expect a lot of unintended side effects. One might want to use STRONG_TYPEDEF to define another int or double type and reset the implementation level of that - but that going off on a tangent. > Actually, my archive initially have only one (non-templated) 'save' for > unsigned. I got compile error until I've declared 'save' for const char*. > I'm not sure why. Its probably because of the above (pointer to a primitive type). Attempts to serialize instances of types whose implementation level is set to not-serializable will result in compile time assertions. (these assertions are sometimes the reason for the deep nested mpl). > > I came to conclude it presented a big security risk. The problem is the > > following: > > > later: > > > > char str[MAX_STRING_SIZE] > > > > ar >> static_cast<char *>(str); // to avoid str being treated as an > array > > > > suppose the text archive gets corrupted to: > > > > 3000 abc............ > > > > The archive will load with a buffer overrun - a security risk. > > Right. I think this problem can be addressed with a wrapper for dynamic > array > > char* str(0); > ar >> make_dynarray_wrapper(str) > > so that the library allocates the string itself. How does it know what size to make the array? Maybe you meen Ar >> make_dynarray_wrapper(str, number of elements * element size) Of course for char arrays one could use Ar << binary_object(str, element_count * element_size) And it would still be portable. > >> Can I define only one 'load' for unsigned int? > > > > As oppose to ? > > As opposed to overloads for all builtin types. For polymorphic_archive we > can't have templated function which 'fallbacks' anywhere. We need to have > a > closed set of virtual functions and I wonder what's the minimal set. > > the rest. BTW this provided a huge benefit. In the original version of > > last year I got into a never-ending battle to specify virtual functions > > which was > > dependent on the compiler - long long, etc. It was hopeless - moving to > > templates solved that. > > :-( I guess we're back to those problems. > I have recently got the polymorphic array working on my machine. Only a couple really small changes to the library code were required. For the list of primitive types I included all portable C++ primitive types. (that is no long long, __int64 etc.) I'm not sure how much interest the polymorphic archive will engender and it's a little hard to understand until you've actually spent enough time with the library to appreciate the limitations of templated code in certain scenarios. So although its clean (and clever) its not going to be easy to explain. Robert Ramey

3 2

[serialization] not so formal review
by Gennadiy Rozental 26 Apr '04

26 Apr '04

Hi, Robert. Unfortunately I was not able to look with proper attention onto your submission during review. One thing I did though: all the major issues I mentioned in my original review looks resolved. I am sure will look into more details later, but for now it is enough for me to cast YES vote. This library cover very important problem domain in software development and I think we all have now solid solution. Thanks and congratulation to Robert for being able to bring it to us. Regards, Gennadiy.

1 0

Serialization Formal Review #2
by Jeff Garland 26 Apr '04

26 Apr '04

All - Today (April 13th) is the start of the formal review of the Serialization library by Robert Ramey. The review will run until Monday April 26th. I will be serving as review manager. Library details are below. I have a number of special requests for this review: 1) This is a large, complex, and important library. Please don't wait till the end of the review period to start your review -- there is no additional room in the review schedule to extend the review period. 2) I am particularly interested in hearing from parties that have tried to extend the library by adding serialization for a new type or by building a new type of archive. The ability to easily extend the library for new types and novel types of archives is critical for this library. 3) If you reviewed the library during the first review, I would like your evaluation of whether the new version of the library addresses your original concerns. Also, if this is a re-review I would appreciate if you could send me a weblink from the mail archive to your original review posting(s). 4) Finally, I would like reviewers to indicate parts of the library that might become standalone boost libraries. While this will have little bearing on the acceptance / rejection of the library it might allow a reduction in the size of the library. As usual, please state in review comments how you reviewed the library and whether the you think the library should be accepted into Boost. Further guidelines for writing reviews can be found on the website at: http://www.boost.org/more/formal_review_process.htm#Comments ********************************************** Library Details: ********************************************** >From the library synopsis: Here, we use the term "serialization" to mean the reversible deconstruction of an arbitrary set of C++ data structures to a sequence of bytes. Such a system can be used to reconstitute an equivalent structure in another program context. Depending on this context, this might used implement object persistence, remote parameter passing or other facility. In this system we use the term "archive" to refer to a specific rendering of this stream of bytes. This could be a file of binary data, text data, XML, or some other created by the user of this library. The latest package can be found at the following locations: http://www.rrsd.com/boost/index.htm http://groups.yahoo.com/group/boost/files/serialization18.zip Thanks, Jeff

5 6

documentation bug report for boost::random
by Dave Gomboc 25 Apr '04

25 Apr '04

The code sample at the top of the documentation is missing a semicolon. > boost::uniform_int<> six(1,6) // distribution that maps to 1..6 > // see random number distributions Dave

1 0

boost:filesystem::path throws eception on valid path name
by Andreas Pokorny 25 Apr '04

25 Apr '04

Hi, $ cat test.cpp #include <stdexcept> #include <iostream> #include <boost/filesystem/path.hpp> int main() { try { boost::filesystem::path p("/usr/lib/gcc-lib/i686-pc-linux-gnu/3.3.3/include/g++-v3/"); } catch( std::exception &e ) { std::cout << e.what() << std::endl; } catch(...) { std::cout << "Unknown exception" << std::endl; } } $ g++ test.cpp -o test -lboost_filesystem $ ./test boost::filesystem::path: invalid name "g++-v3" in path: "/usr/lib/gcc-lib/i686-pc-linux-gnu/3.3.3/include/g++-v3/" Is that supposed to happen? Regards, Andreas Pokorny

2 1

Quick thought on Date-Time
by Daryle Walker 25 Apr '04

25 Apr '04

[NOTE: I haven't worked on this library; I'm just going by the posts I see here.] For this increment/decrement by a month problem, is there any way to store a date with only a month resolution? Nothing smaller will be stored, so day crossovers won't be a problem. Wouldn't something similar have/already-had been done when skipping days but ignoring the exact hour? (Or does every date-time object work at the fractional-second level? If so, maybe that's what needs fixing.) -- Daryle Walker Mac, Internet, and Video Game Junkie darylew AT hotmail DOT com

4 9

Re: [boost] [serialization+date_time] Serialization Review: serialization of dates
by JOAQUIN LOPEZ MU?Z 25 Apr '04

25 Apr '04

----- Mensaje original ----- De: Jeff Garland <jeff(a)crystalclearsoftware.com> Fecha: Domingo, Abril 25, 2004 11:19 pm Asunto: Re: [boost] [serialization+date_time] Serialization Review: serialization of dates [...] > > #include "boost/archive/basic_archive.hpp" > > #include "boost/serialization/split_free.hpp" > > #include "boost/serialization/string.hpp" > > > > I think the two first are unnecessary (could you > > try this out? thx) > > Actually none of them are necessary. Ah the wonder of templates :-) > Wonderful they are, you're right :) So, I guess this is as good as it can get. Slightly OT: this issue of POI could be taken a little further to allow interoperability between (not necessarily template-based) libraries without the headache of having to include portions of them. This is what I have in mind: suppose a header-based library A provides support for or somehow uses a non-header (i.e. requires linking) library B, like this: // lib A #include <b.hpp> inline foo(){...} // does not depend on B inline bar(){...} // does not depend on B inline mess_with_B(){...} Now, if B is not linked in the project mess_with_B won't compile, so one has to move this function to a separate header that the user will/won't include depending of whether she wishes that functionality. Instead, one can do the following: // lib A #include <b.hpp> inline foo(){...} inline bar(){...} template<typename dummy> inline mess_with_B(dummy* =0){...} Voila, A works even if B is not linked in. If the user has linked B and wants to use mess_with_B, then she just has to call mess_with_B() as usual, template instantiation model will do the rest. Maybe I'm missing something, but I think this approach could be investigated a little further. One good example of use is for interoperability with Boost.Threads, for instance. Totally OT: I've posted a question regarding multi_index and composite keys a few posts ago. You being one of the reviewers, your opinion on that is greatly appreciated :) Joaquín M López Muñoz Telefónica, Investigación y Desarrollo

2 1

Re: [boost] [serialization+date_time] Serialization Review: serialization of dates
by JOAQUIN LOPEZ MU?Z 25 Apr '04

25 Apr '04

----- Mensaje original ----- De: Jeff Garland <jeff(a)crystalclearsoftware.com> Fecha: Domingo, Abril 25, 2004 9:25 pm Asunto: Re: [boost] [serialization+date_time] Serialization Review: serialization of dates > On Sun, 25 Apr 2004 20:34:34 +0200, JOAQUIN LOPEZ MU?Z wrote > > > Regarding the need to include "greg_serialize.hpp", I think > > a better approach, from the point of view of the user, is > > that serialization support is already embedded into > > "boost/date_time/gregorian/gregorian_types.hpp" and > > automatically provided if one of the archive headers > > > >...snip details... > > > > See what I mean? For this to work with minimum > > overhead, we'd need some forward headers for serialize, > > save, load, etc. to be included by > > "boost/date_time/gregorian/gregorian.hpp". > > Yes, I agree on the user perspecitive. I'm sorry I forgot to say > that in the > first few emails. I did say it here: > http://lists.boost.org/MailArchives/boost/msg63885.php > > > I'm not sure whether this forward headers are already > > provided by Robert's library. > > Why would I need 'forward headers' for this? > Well, since serialization support is entirely contained in function templates, you don't need any actual definition till POI (I might be wrong here, but I guess I'm not.) This can help reduce compilation time if you're not going to serialize dates at all. Matter of fact, seeing the includes in your example #include "boost/archive/basic_archive.hpp" #include "boost/serialization/split_free.hpp" #include "boost/serialization/string.hpp" I think the two first are unnecessary (could you try this out? thx) Joaquín M López Muñoz Telefónica, Investigación y Desarrollo

2 1