- Boost - lists.boost.org

Re:Re: [boost] Serialization Formal Review #2
by Robert Ramey 27 Apr '04

27 Apr '04

Matthias Troyer wrote: > I have tried extending the library since I want to swap our old > serialization library used at http://alps.comp-phys.org with a Boost > serialization library as soon as possible. It was important to keep the > same archive format, since we have gigabytes of data collected over ten > years in millions of CPU-hours and need to be able to read those with > both new and old codes. With the help of Robert I managed to convince > myself that this is indeed possible Perhaps it might be possible, but I'm doubtful that it's the best way to address the problem. If the legacy format is a meta-data format it can be possible in a way similar to the way the XML was handled. But in general it will not be desirable nor necessary. To see this take an extreme example. Suppose over the last 10 years we had 10 different programmers working on a the project. Each one had to save and reload data for his classes. Now we want to convert to the new system. We would need a new archive class that tracked all the idiosyncracies in the current code. A huge job. And there exists a much simpler approach. Loading legacy data(legacy_data & ld) ==================== Ifstream is("old_data"); // read first line of old_data file // if it doesn't have the serialization signature // load the data using the legacy system. else{ Boost::binary_iarchive ia(is); ia >> ld } Save data // save in using serialization. As file are processed they are automatically converted to the new system. No special programming is required as the code to load the class instances from the legacy data format already exists - it's the legacy code ! its free at this point. > 4. Documentation of archive formats, especially what class and object > information is stored. I want to be able to predict, from reading the > documentation, what exactly will be written to the archive and in which > order. This would entail a detail paraphrasing of the operation of the code. In the case where there is no versioning, no tracking and no pointer serialization its fairly simple. When any of the others are present, its starts to get pretty complex. There is a comment in basic_archive.cpp which in fact summarizes the format - but I'm not convinced it something that belongs in user documentation. ////////////////////////////////////////////////////////////////////// // // class_information is stored as // // class_id* // -1 for a null pointer // if a new class id // [ // exported key - class name* // tracking level - always/never // file version // ] // // if tracking // [ // object_id // ] // // [ // if a new object id // data... // ] // // * required only for pointers - optional for objects This recursively defines the file format for any serialized data structure. > This is essential when exchanging data with an application that > does not use this library. I'm skeptical that this is going to be fruitful. The "other" application is going to have to effectively re-implement this whole library. Why bother, just use this one. To send data to "other" applications, there is always the XML archive. I don't think that a general purpose tool such as serialization is going to be very helpful in trying to implement some externally defined data format. Some meta-data formats (e.g. XML and windows ini files) are doable but in general it's not going to be productive. Robert Ramey

2 1

Type traits question
by Robert Ramey 26 Apr '04

26 Apr '04

Is there a type traits guru that can explain to me why the following doesn't work as I expect. It's got to be something very, very obvious but I can't see it. Thank you Robert Ramey #include <boost/static_assert.hpp> #include <boost/type_traits/is_base_and_derived.hpp> #include <boost/type_traits/remove_pointer.hpp> class Base1 { char a; }; class Base2 { int b; }; class Derived : public Base1, public Base2 { long c; }; void main(){ // this passes BOOST_STATIC_ASSERT((boost::is_base_and_derived< Base1, Derived >::value)); // this fails !? BOOST_STATIC_ASSERT((boost::is_base_and_derived< boost::remove_pointer<Base1 *>, boost::remove_pointer<Derived *> >::value)); }

3 2

RE: Type traits question
by Robert Ramey 26 Apr '04

26 Apr '04

Never mind, I see it now. Sorry I was desperate. Robert Ramey Robert Ramey wrote: > > #include <boost/static_assert.hpp> > #include <boost/type_traits/is_base_and_derived.hpp> > #include <boost/type_traits/remove_pointer.hpp> > > class Base1 > { > char a; > }; > > class Base2 > { > int b; > }; > > class Derived : public Base1, public Base2 > { > long c; > }; > > void main(){ > // this passes > BOOST_STATIC_ASSERT((boost::is_base_and_derived< > Base1, > Derived > >::value)); > > // this fails !? > BOOST_STATIC_ASSERT((boost::is_base_and_derived< > boost::remove_pointer<Base1 *>, > boost::remove_pointer<Derived *> > >::value)); > }

1 0

Re: [boost] [multi_index] composite keys: request for advice
by JOAQUIN LOPEZ MU?Z 26 Apr '04

26 Apr '04

----- Mensaje original ----- De: Rob Stewart <stewart(a)sig.com> Fecha: Lunes, Abril 26, 2004 10:16 pm Asunto: Re: [boost] [multi_index] composite keys: request for advice > From: Joaquin M Lopez Munoz <joaquin(a)tid.es> > > > > Composite keys can be specified by a new construct > > called composite_key in the following manner: > > > > struct record > > { > > int first; > > int second > > int third; > > }; > > > > multi_index_container< > > record, > > indexed_by< > > composite_key< > > record, //base value > > member<record,int,&record::first>, // first key > > Would that you could make this work to avoid the duplication and > chance for errors: > > member< &record::first > > > I know a function template can deduce the other types, but I > can't think of how to make it work here. > I'd be the happiest man on the block if I could have that, but AFAIK it is not possible. The fact that member<> is here specified in the context of composite_key does not help, either. This has been previously brought into discussion several times. If you have a typeof() utility and don't object to some macro hackery, this is an approximation: template<typename Class,typename Member> Class base_from_ptr_to_member(Member Class::*); template<typename Class,typename Member> Member member_from_ptr_to_member(Member Class::*); #define BOOST_MULTI_INDEX_SMART_MEMBER(PtrToMember) \ member< \ typeof(base_from_ptr_to_member(PtrToMember)),\ typeof(member_from_ptr_to_member(PtrToMember)),\ PtrToMember\ > ... multi_index_container< record, indexed_by< ordered_unique< BOOST_MULTI_INDEX_SMART_MEMBER(&record::x) > > > Joaquín M López Muñoz Telefónica, Investigación y Desarrollo

1 0

[multi_index] composite keys: request for advice
by Joaquin M Lopez Munoz 26 Apr '04

26 Apr '04

> I'm working through the post review changes of multi_index and hope to have something released in one or two weeks. One of the issues raised in the review was the possibility to promote what was an example of composition of keys to be part of the library. I've come up with something more ambitious, but there's a design decision of which I'm not quite certain wich way to go, and I'd like to ask you for advice. Composite keys can be specified by a new construct called composite_key in the following manner: struct record { int first; int second int third; }; multi_index_container< record, indexed_by< composite_key< record, //base value member<record,int,&record::first>, // first key member<record,int,&record::second>, // 2nd member<record,int,&record::third> // 3rd > > > mc; In the example the composite key is composed of three key extractors, and the sorting is made in lexicographical order, for instance: 0,0,1 0,0,2 0,1,0 0,5,4 1,0,0 1,0,2 1,2,3 ... Given the lexicographical order, it is possible to efficiently search for incomplete keys where only the first values are given: mc.equal_range(make_tuple(1,0)) yields 1,0,0 1,0,2 mc.equal_range(make_tuple(0)) yields 0,0,1 0,0,2 0,1,0 0,5,4 and so on. Accordingly, comparison operators between composite_key results and tuples are overloaded so that the following holds: composite_key</* as before*> ck; ck(record(1,2,3))<=make_tuple(1,2); // yields true make_tuple(1,2)<=ck(record(1,2,3)); // yields true So far, this is all IMHO perfectly consistent, but the following design decision is not obvious to me: given the previous, what should be the result of ck(record(1,2,3))==make_tuple(1,2); // true or false? As these two objects are neither greater than the other, at least they are *equivalent*, but maybe allowing operator== to return true is too much. An argument against making them equal is that, with future hashed indices in mind, hashing has to be done on the whole tuple of values returned by the composite key so that: multi_index_container< record unordered_unique< //hashed composite_key< /* as before */ > > > mc; mc.find(make_tuple(1,2,3)); //OK mc.find(make_tuple(1,2)); //KO, we need the whole tuple If we make ck(record(1,2,3))==make_tuple(1,2) return true, the impossibility of having the last line to work is puzzling. If we make it return false, it is also puzling that neverthelss operator<= is true in both senses. What do I do? a. require operator== between composite_key results and tuples to work only on full tuples. b. define operator== in the same line operator<, <= etc. are defined. Thanks for your comments. Joaquín M López Muñoz Telefónica, Investigación y Desarrollo

5 4

RE: Re: Re:[boost] Re: Serialization Formal Review #2
by Rozental, Gennadiy 26 Apr '04

26 Apr '04

> -----Original Message----- > From: boost-bounces(a)lists.boost.org > [mailto:boost-bounces@lists.boost.org] On Behalf Of Robert Ramey > Sent: Monday, April 26, 2004 2:02 PM > To: boost(a)lists.boost.org > Subject: Re:Re: Re:[boost] Re: Serialization Formal Review #2 > > > Peter Dimov wrote: > > > FWIW, in my own library I do, indeed, have a special case code for > > serializing an unmodified shared_ptr, for two reasons. First, this > > shows that the library is flexible enough. Second, even if > we modify > > boost::shared_ptr to be boost::serialization-friendly, we won't be > > able to patch std::tr1::shared_ptr. > > The main problem I had was that certain member variable > weren't accessible. The STL collections don't have that issue > so there was no problem. Sorry if this is silly question, but why would you need an access to the counter? IMO we should be able to save/load shared_ptr without need to save it. Gennadiy.

1 0

RE: [boost] Re: Serialization Formal Review #2
by Rozental, Gennadiy 26 Apr '04

26 Apr '04

> > Your library cannot be used to save/load anything before main() is > > entered nor can it be used to save/load anything after > main() has been > > left. > > Correct. It never occurred to me to mention this. > > Robert Ramey It probably could be addressed by some kind of lifetime management scheme. Essentially you need to make sure that before any serialization code is invoked all static initializes gets instantiated. I am not sure it doable since you actually within the library code may not know what should be instantiated. On the other hand I an not sure that any move in this direction even worth the trouble. I guess it may be convenient in some cases. Gennadiy.

1 0

Re:Re: Re:[boost] Re: Serialization Formal Review #2
by Robert Ramey 26 Apr '04

26 Apr '04

Peter Dimov wrote: > FWIW, in my own library I do, indeed, have a special case code for > serializing an unmodified shared_ptr, for two reasons. First, this shows > that the library is flexible enough. Second, even if we modify > boost::shared_ptr to be boost::serialization-friendly, we won't be able to > patch std::tr1::shared_ptr. The main problem I had was that certain member variable weren't accessible. The STL collections don't have that issue so there was no problem. > However, should you decide to use your current scheme, I'll happily insert > whatever friend declarations are necessary in shared_count. > I'm away from my computer right now, so I can't paste my read/write > functions for shared_ptr, but the general idea is that they keep a > map< shared_ptr<void>, int > on writing, mapping every shared_ptr > ownership group to a pointer id, and a corresponding map< int, > shared_ptr<void> > on reading, for the reverse transformation. > The main problem for implementing a non-intrusive serialize() for > shared_ptr is that these pointer maps need to be held in the archive, > which requires either archive modification, or a general "extra state" > support in all archives. That's pretty much the same as what the serialization system does - albeit in perhaps in a more generic way. Its not clear to me how enhanced library code could be made to work work without granting access to shared_count. Of course, if access is granted to shared_count, then everything works already. Robert Ramey

1 0

ReL[boost] Re: Serialization Formal Review #2
by Robert Ramey 26 Apr '04

26 Apr '04

David Abrahams wrote: >"Peter Dimov" <pdimov(a)mmltd.net> writes: >>> http://www.boost.org/libs/smart_ptr/intrusive_ptr.html#Introduction >>> >>> However, the situation is different there as the called function >>> only accepts exactly one parameter. >> >> It doesn't matter. ADL should find the overload, provided that >> 'serialize' >> is not qualified at the point of the call. > >> In my own library I use very similar read/write overloads that are, >> indeed, placed in the namespace of the corresponding class and found by >> ADL. Yes, it works with VC 7.1. >The recommended "overload in the library namespace" procedure was a >big concern for me, but I didn't post anything because I couldn't >remember why it's an issue. I have in my mind, left over from my work >with Peter on this problem, a rule that in the current language you >shouldn't do that, and should rely instead on ADL. I don't know why >I think that anymore, but I *can* say that Peter should be listened >to in this area; he's more aware of the issues than almost anyone. OK, I've looked at code and found that found that my calls to "serialize" are qualified by boost::serialization. So this can be fixed. It seems that users of older compilers will have to overload in the library namespace but that it will not be necessary for users that are only concerned with newer compilers. So its not a problem for the library. Except... What about when I've overloaded from with in the library - such as when default serialization traits are overloaded. Should this namespace specification be #ifdef/#endif depending on the compiler config? Should this be dependent on BOOST_NO_ARGUMENT_DEPENDENT_LOOKUP? Robert Ramey

1 0

Re:[boost] Re: Serialization Formal Review #2
by Robert Ramey 26 Apr '04

26 Apr '04

Andreas Huber wrote: > Ok, but IIUC this undermines the original promise that a class > implementing serialize or safe/load can be serialized to *any* archive, > even future ones without changing its implementation. I see that I > wouldn't need to change the serialization *code* but I'd need to add > an include to the .cpp file, right? Correct. But I don't see that as undermining the original promise. You will have to re-compile for future archives, but code will not have to be changed. The polymorphic archive (in progress) will lift even that restriction. > > >> 9. The library registers all kinds of stuff with static instances. > >> This precludes its use outside main(). > > > Hmmm - that would be news to me. > Your library cannot be used to save/load anything before main() is entered > nor can it be used to save/load anything after main() has been left. Correct. It never occurred to me to mention this. Robert Ramey

1 0