- Boost - lists.boost.org

[multi_index] composite keys: request for advice
by Joaquin M Lopez Munoz 26 Apr '04

26 Apr '04

> I'm working through the post review changes of multi_index and hope to have something released in one or two weeks. One of the issues raised in the review was the possibility to promote what was an example of composition of keys to be part of the library. I've come up with something more ambitious, but there's a design decision of which I'm not quite certain wich way to go, and I'd like to ask you for advice. Composite keys can be specified by a new construct called composite_key in the following manner: struct record { int first; int second int third; }; multi_index_container< record, indexed_by< composite_key< record, //base value member<record,int,&record::first>, // first key member<record,int,&record::second>, // 2nd member<record,int,&record::third> // 3rd > > > mc; In the example the composite key is composed of three key extractors, and the sorting is made in lexicographical order, for instance: 0,0,1 0,0,2 0,1,0 0,5,4 1,0,0 1,0,2 1,2,3 ... Given the lexicographical order, it is possible to efficiently search for incomplete keys where only the first values are given: mc.equal_range(make_tuple(1,0)) yields 1,0,0 1,0,2 mc.equal_range(make_tuple(0)) yields 0,0,1 0,0,2 0,1,0 0,5,4 and so on. Accordingly, comparison operators between composite_key results and tuples are overloaded so that the following holds: composite_key</* as before*> ck; ck(record(1,2,3))<=make_tuple(1,2); // yields true make_tuple(1,2)<=ck(record(1,2,3)); // yields true So far, this is all IMHO perfectly consistent, but the following design decision is not obvious to me: given the previous, what should be the result of ck(record(1,2,3))==make_tuple(1,2); // true or false? As these two objects are neither greater than the other, at least they are *equivalent*, but maybe allowing operator== to return true is too much. An argument against making them equal is that, with future hashed indices in mind, hashing has to be done on the whole tuple of values returned by the composite key so that: multi_index_container< record unordered_unique< //hashed composite_key< /* as before */ > > > mc; mc.find(make_tuple(1,2,3)); //OK mc.find(make_tuple(1,2)); //KO, we need the whole tuple If we make ck(record(1,2,3))==make_tuple(1,2) return true, the impossibility of having the last line to work is puzzling. If we make it return false, it is also puzling that neverthelss operator<= is true in both senses. What do I do? a. require operator== between composite_key results and tuples to work only on full tuples. b. define operator== in the same line operator<, <= etc. are defined. Thanks for your comments. Joaquín M López Muñoz Telefónica, Investigación y Desarrollo

5 4

RE: Re: Re:[boost] Re: Serialization Formal Review #2
by Rozental, Gennadiy 26 Apr '04

26 Apr '04

> -----Original Message----- > From: boost-bounces(a)lists.boost.org > [mailto:boost-bounces@lists.boost.org] On Behalf Of Robert Ramey > Sent: Monday, April 26, 2004 2:02 PM > To: boost(a)lists.boost.org > Subject: Re:Re: Re:[boost] Re: Serialization Formal Review #2 > > > Peter Dimov wrote: > > > FWIW, in my own library I do, indeed, have a special case code for > > serializing an unmodified shared_ptr, for two reasons. First, this > > shows that the library is flexible enough. Second, even if > we modify > > boost::shared_ptr to be boost::serialization-friendly, we won't be > > able to patch std::tr1::shared_ptr. > > The main problem I had was that certain member variable > weren't accessible. The STL collections don't have that issue > so there was no problem. Sorry if this is silly question, but why would you need an access to the counter? IMO we should be able to save/load shared_ptr without need to save it. Gennadiy.

1 0

RE: [boost] Re: Serialization Formal Review #2
by Rozental, Gennadiy 26 Apr '04

26 Apr '04

> > Your library cannot be used to save/load anything before main() is > > entered nor can it be used to save/load anything after > main() has been > > left. > > Correct. It never occurred to me to mention this. > > Robert Ramey It probably could be addressed by some kind of lifetime management scheme. Essentially you need to make sure that before any serialization code is invoked all static initializes gets instantiated. I am not sure it doable since you actually within the library code may not know what should be instantiated. On the other hand I an not sure that any move in this direction even worth the trouble. I guess it may be convenient in some cases. Gennadiy.

1 0

Re:Re: Re:[boost] Re: Serialization Formal Review #2
by Robert Ramey 26 Apr '04

26 Apr '04

Peter Dimov wrote: > FWIW, in my own library I do, indeed, have a special case code for > serializing an unmodified shared_ptr, for two reasons. First, this shows > that the library is flexible enough. Second, even if we modify > boost::shared_ptr to be boost::serialization-friendly, we won't be able to > patch std::tr1::shared_ptr. The main problem I had was that certain member variable weren't accessible. The STL collections don't have that issue so there was no problem. > However, should you decide to use your current scheme, I'll happily insert > whatever friend declarations are necessary in shared_count. > I'm away from my computer right now, so I can't paste my read/write > functions for shared_ptr, but the general idea is that they keep a > map< shared_ptr<void>, int > on writing, mapping every shared_ptr > ownership group to a pointer id, and a corresponding map< int, > shared_ptr<void> > on reading, for the reverse transformation. > The main problem for implementing a non-intrusive serialize() for > shared_ptr is that these pointer maps need to be held in the archive, > which requires either archive modification, or a general "extra state" > support in all archives. That's pretty much the same as what the serialization system does - albeit in perhaps in a more generic way. Its not clear to me how enhanced library code could be made to work work without granting access to shared_count. Of course, if access is granted to shared_count, then everything works already. Robert Ramey

1 0

ReL[boost] Re: Serialization Formal Review #2
by Robert Ramey 26 Apr '04

26 Apr '04

David Abrahams wrote: >"Peter Dimov" <pdimov(a)mmltd.net> writes: >>> http://www.boost.org/libs/smart_ptr/intrusive_ptr.html#Introduction >>> >>> However, the situation is different there as the called function >>> only accepts exactly one parameter. >> >> It doesn't matter. ADL should find the overload, provided that >> 'serialize' >> is not qualified at the point of the call. > >> In my own library I use very similar read/write overloads that are, >> indeed, placed in the namespace of the corresponding class and found by >> ADL. Yes, it works with VC 7.1. >The recommended "overload in the library namespace" procedure was a >big concern for me, but I didn't post anything because I couldn't >remember why it's an issue. I have in my mind, left over from my work >with Peter on this problem, a rule that in the current language you >shouldn't do that, and should rely instead on ADL. I don't know why >I think that anymore, but I *can* say that Peter should be listened >to in this area; he's more aware of the issues than almost anyone. OK, I've looked at code and found that found that my calls to "serialize" are qualified by boost::serialization. So this can be fixed. It seems that users of older compilers will have to overload in the library namespace but that it will not be necessary for users that are only concerned with newer compilers. So its not a problem for the library. Except... What about when I've overloaded from with in the library - such as when default serialization traits are overloaded. Should this namespace specification be #ifdef/#endif depending on the compiler config? Should this be dependent on BOOST_NO_ARGUMENT_DEPENDENT_LOOKUP? Robert Ramey

1 0

Re:[boost] Re: Serialization Formal Review #2
by Robert Ramey 26 Apr '04

26 Apr '04

Andreas Huber wrote: > Ok, but IIUC this undermines the original promise that a class > implementing serialize or safe/load can be serialized to *any* archive, > even future ones without changing its implementation. I see that I > wouldn't need to change the serialization *code* but I'd need to add > an include to the .cpp file, right? Correct. But I don't see that as undermining the original promise. You will have to re-compile for future archives, but code will not have to be changed. The polymorphic archive (in progress) will lift even that restriction. > > >> 9. The library registers all kinds of stuff with static instances. > >> This precludes its use outside main(). > > > Hmmm - that would be news to me. > Your library cannot be used to save/load anything before main() is entered > nor can it be used to save/load anything after main() has been left. Correct. It never occurred to me to mention this. Robert Ramey

1 0

Re:[boost] Re: Formal review: serialization
by Robert Ramey 26 Apr '04

26 Apr '04

Jeff F wrote: > template<class Archive> > inline void serialize( > Archive& ar > , boost::filesystem::path& t > , unsigned int version ) > { > std::string lPathString; > > if( is_saving(ar) ) > lPathString = t.native_file_string(); > > ar & make_nvp( "PathString", lPathString ); > > if( is_loading(ar) ) > t = boost::filesystem::path( lPathString, > boost::filesystem::native ); > } The "real" solution here is to implement serialization for "native_file_string() So that things look like template<class Archive> void serialize( Archive &ar, boost::filesystem::native_files_sting_type & t, unsigned int version ){ ... ?; } template<class Archive> void serialize( Archive &ar, boost::filesystem::path & t, unsigned int version ){ ar & make_nvp("PathString", t); } Robert Ramey

1 0

Re:[boost] Re:Re:Re:Formal review: serialization
by Robert Ramey 26 Apr '04

26 Apr '04

Vladimir Prus wrote: >Hmm... gotta look into this. What about my second question: >>> 2. For random access I need to make sure that all saved objects have >>> export key. How do I do that? Not necessary out-of-the box but where >>> I can plug the check? Actually, its not necessary that ALL saved objects have export keys. Only those objects that are serialized through polymorphic pointers need to be exported or registered. An attempt to serialize a polymorphic pointer that has neither been exported nor explicitly registered will trigger a run-time exception "unregistered_class". If you wanted to enforce the idea that all types should be exported I believe it would be possible to something like the following in your archive class: a) override implementation of save(class_id_type & t) and save(class_id_optional_type) so that they set a flag. b) override implementation of save(class_name & t) it sets a flag c) override end_preamble so that it checks for an undesired set of flags. I never considered this before so you'll have to check the above yourself. >>> one for dynamic arrays with element-wise save >> >> I would envision one using >> >> ar << element_count >> For(i = 0; I < element_count; i++) >> ar << element[i] >> >> I'm not sure that its worth adding and documenting such an obvious and >> trivial thing as a separate wrapper. >But on loading you need to add 'new[]' call. So load and save become >non-symmetric, and you need to split serialize, which is rather >inconvenient. Actually, I forget the original question. But in any case, I'm wondering about trying to anticipate every application. OTOH, having considered again dynamic_array I'm now more inclined to include it. I'll take a look a the other ones. These would be described in the document under "serialization wrappers". > > There are cases when an object id is > > assigned but not written to the archive I don't see these as having much > > utility outside of serialization. > So, it's not easily possible to plug a different algorithm? In a general sense - NO. Any of the "bookkeeping" types in common_?archive can be overloaded to taste. Overloading these in a special way was how XML was implemented. I'm sure that these could be used for creating other special purpose archives but I really can't know for any particular case until one tries to implement them. >>> Ah, I've missed that. Do I need to provide both 'type' and 'value'? >>> Can't serialization library work with just one? > >> Could be. I just provided both so that I could interoperate with mpl >> without having to think about each specific case. > But for user this can be inconvenient. Actually I was thinking most users would use the macros. If one fails to include one that's used it should fail compilation so I don't see it's a show stopper. Actually I think it's easier just to always include both rather than trying to document/remember which ones use which. > It it possible that you make BOOST_EXPORT always > register classes with the polymoprhic archive. > and use polymoprhic archive as fallback for serializing classes? I'm still fine tuning this (smart_cast anyone?). But basically things work like this. a) Just as now, one could pre-compile a serialization instantiating code for particular archives. In this case one of the archives would be the "polymorphic_?archive" class which is nothing but a bunch of virtual function declarations. b) when the application is built, a specific archive implemenatation is selected. E.G polymorphic_text_?archive. The is archive inherits the implementation from text_?archive and the interface from polymorphic_?archive. c) the application will link with the pre-compiled classes. I'm not sure this answers your question - but there it is. Thank god no ones asked about DLL implementations yet. Robert Ramey

1 0

Re:[boost] Re: Serialization Formal Review #2
by Robert Ramey 26 Apr '04

26 Apr '04

Message: 11 Andreas Huber wrote: > 1. On compilers supporting ADL, it seems the user-supplied serialization > function for non-intrusive serialization > template<class Archive> > void serialize(Archive & ar, gps_position & g, const unsigned int > version); > could also be put into the namespace of the type that it serializes. If > this is correct it should be mentioned in the tutorial & reference. If > not, there should be a rationale why not. I did try this in my environment VC 7.1 and it failed to work. Does anyone want to assert that it should or shouldn't work? > 2. save/load naming: All functions that have to do with reading from an archive are named load*, > all functions that have to do with writing to an archive are named save*. > To me (non-native speaker) these names are generally associated with > persistence only. Since the library is more general (one could use it for > network communication, etc.) load/save are a bit missleading. I'd rather > see read/write or other names that more clearly communicate what the > functions are doing. I used save/load consistently specifically to distinguish from read/write which seemed to me too suggestive of persistence, files and streams. > 3. archive naming: > The word archive also seems to be associated with persistence. I think > serializer would much better describe the functionality. Another option I > could live with is serialization_stream (I slightly disagree with the > assertion in archives.html that an archive as modelled by this library is > not a stream. It might not be what C++ folks traditionally understand a > stream is but to me it definitely is a more general stream, i.e. a stream > of objects/things/whatever). Archive is perhaps a little too suggestive. I borrowed from MFC. Making changes would have a huge ripple effect through code, file names, namepace names and documentation. I don't find any proposals for alternate naming sufficiently compelling to justify this. > 4. Object tracking 1: Seeming contradiction > I've had difficulty determining how exactly object tracking works. I > believe the associated docs are slightly contradictory: > - Under exceptions.html#pointer_conflict I read that the pointer_conflict > exception is thrown when we save a class member through a pointer first > and then "normally" (i.e. through a reference). Correct. > - traits.html#tracking says "... That is[,] addresses of serialized > objects are tracked if and only if an object of the same type is anywhere > in the program serialized through a pointer." Correct. > The second sentence seems to contradict the first one in that it does not > specify that the sequence of saving is important I see no conflict here > (I assume that pointer_conflict exception is not thrown if we save > normally first and then through a pointer). Correct > Moreover, I don't see how the library could possibly > achieve the "anywhere in the program" bit. It is tricky and it does work. I works by instantitiating a static variable. Static variables are constructed before main() is invoked. These static variables register themselves in global table when constructed. So even though a template is used "anywhere in the program" the entry in the global table exists before execution starts. I'm reluctant to include such information in the manual. > 5. Object tracking 2: It should not be possible to serialize pointers > referencing objects of types that have track_never tracking type. LOL - tell that to library users. There are lots of things that users are convinced they have to do that I don't think are a good idea. As a library writer I'm not in position to enforce my ideas how users should use the library. On this particular point I would be very suspicious about such a usage but as practical matter I just have to include a warning and move on. special.html#objecttracking says: <quote> By definition, data types designated primitive by Implementation Level class serialization trait are never tracked. If it is desired to track a shared primitive object through a pointer (e.g. a long used as a reference count), It should be wrapped in a class/struct so that it is an identifiable type. The alternative of changing the implementation level of a long would affect all longs serialized in the whole program - probably not what one would intend. </quote> > I think serializing untracked pointers almost never makes sense. If it > does anyway the pointed-to object can always be saved by dereferencing the > pointer. Such an error should probably be signalled with an exception. I don't understand this. > 6. Object tracking 3: One should be able to activate tracking on the fly > Rarely it makes sense to save even primitive types in tracked mode (e.g. > the example quoted in 6.). That's why I think setting the tracking type > per type is not flexible enough. It should be possible to override the > default somehow. I could imagine to save such pointers as follows: > class T > { > ... > int i; > int * pI; // always points to i > }; > template<class Archive > > void T::save(Archive &ar) const > { > ar << track( i ); // save the int here, keep track of the pointer > ar << track( pI ); // only save a reference to the int > } > Since it is an error to save an untracked type by pointer even the second > line needs the track( ) call. A main function of using a trait is to avoid instantiation of code that won't ever be used. If I understand your proposal we would have to instantiate tracking code even if it isn't used. But my main objection thing about this proposal is that it introduces a runtime modality into the type traits. This sound like it would very hard to debug and verify. I see lots of difficulties and very little advantage to trying to add this. > 7. I think the enum tracking_type should rather be named tracking_mode as > the _type suffix suggests something else than an enum. I don't think the change is sufficiently compelling to justify the hassle associated with this. > 8. I don't understand how function instantiation as described under > special.html#instantiation works. The description seems to indicate that > all possible archive classes need to be included in the files of all types > that are exported. Is that true? True. All archives that might be used have to be included. This is an artifact of using templates rather than virtual functions. Templates regenerate the code for each combination of serializable type and archive. This results in faster code - but more of it. There were strong objections raised in the first review about the fact it used virtual functions rather then templates - so it was changed to templates here. > 9. The library registers all kinds of stuff with static instances. This > precludes its use outside main(). Hmmm - that would be news to me. Robert Ramey

6 7

Re: [boost] [serialization] not so formal review
by Robert Ramey 26 Apr '04

26 Apr '04

Gennadiy Rozental wrote > Unfortunately I was not able to look with proper attention onto your > submission during review. One thing I did though: all the major issues I > mentioned in my original review looks resolved. I am sure will look into > more details later, but for now it is enough for me to cast YES vote. > This library cover very important problem domain in software development > and I think we all have now solid solution. Thanks and congratulation to > Robert for being able to bring it to us. I was waiting for a comment from you. Your incredibly insightful, exhaustive and focused review of the previous version was the main driving force behind this version. Looking back, it's still hard to for me to believe that in the six hours you said you spent anyone could really understand the implementation to the depth you did. Basically, all I did was to implement all your suggestions one at a time. Early on I switched over to your test library. This permitted me to add features one at a time and test at each stage. It did result in a lot of code being rewritten several times. But I think in system like this one can't really look far enough into the future implementation to avoid this. So it's not too much to say that if this library lives up to the hopes and expectations I have for it, it will be due to a collision between your previous review and my personality flaw that inspires me to throw myself at lost causes. I was going to suggest you might separate some time to review it. On the other hand, maybe I should quite while I'm ahead. Robert Ramey

1 0