[boost] Re:Re:Re:Formal review: serialization

From: Vladimir Prus (ghost_at_[hidden])
Date: 2004-04-26 02:58:33

Robert Ramey wrote:

>> I guess I have two questions:
>> 1. Won't serialization fail in some way if I just seek the stream to the
>> position found in index and try reading.
>> 2. For random access I need to make sure that all saved objects have
>> export
>> key. How do I do that? Not necessary out-of-the box but where I can plug
>> the check?

> Random access into an archive would require some thought.  First, it's not
> clear how it would be used.  In general archives are nested structures,
> one could de-serialize an inner piece - but to where? The original data
> structure was embedded in something else which is not now there.  So this
> can really only consider in the context of a specific application.  Here
> are a couple of plausible scenarios.
> 
> a) A general purpose archive browser - this could browse the archive in a
> random manner but wouldn't actually de-serialize the data.  I don't see
> any
> real problem here.  One would need to create an index either as a side
> effect of archive create or with a second pass over the final archive.

I'd rather like to deserialize the data when needed.

> 
> b) using serialization for a data-state logging.
> 
> Log(Archive ar, statedata){
> // save seek point to index
> // append to archive
> ar << statedata;
> }
> 
> Recover(Archive ar, statedata &, seekpoint)
> // set stream seekpoint
> ar >> statedata
> }
> 
> I could envision something like this being made to work.

Yes, I look for something like this. 
> 
> So I would say that generally, serialization would be expected to be a
> serial operation. (surprise!).   On the other hand, in certain special
> situations it might be possible/convenient to provide for some random
> access but I would expect that to be application specific.

Hmm... gotta look into this. What about my second question:
>> 2. For random access I need to make sure that all saved objects have
>> export
>> key. How do I do that? Not necessary out-of-the box but where I can plug
>> the check?

>> one for dynamic arrays with  element-wise save
> 
> I would envision one using
> 
> ar << element_count
> For(i = 0; I < element_count; i++)
> ar << element[i]
> 
> I'm not sure that its worth adding and documenting such an obvious and
> trivial thing as a separate wrapper.

But on loading you need to add 'new[]' call. So load and save become
non-symmetric, and you need to split serialize, which is rather
inconvenient.

>> and another for dynamic arrays with binary save.
> 
> I see the binary object as filling that role.
> 
> Ar << make_binary_object(dynamic_array_address, element_size *
> element_count);
> 
> I once did propose a manual section name "Serialization Wrappers" which
> would have a couple of examples and highlight the fact that nvp and
> binary_object are instances of this concept.  The idea was received
> unenthusiastically at the time but now I'm becoming convinced It's more
> effective to explain and illustrate the concept rather than try to
> anticipate all the wrappers that users might need.  Actually, I think
> having such a section would have avoided the confusion surrounding the
> intended usage and purpose of binary_object.

Sure, docs rarely hurt.

>> I agree. I actually have a *crazy* but cute idea that one can use file
>> offset for object id. How object ids are assigned and can I customize
>> that process? That would keep overhead at absolute minimum.
> 
> Object_id are assigned sequentially starting with 0.  They are only used
> for classes which require tracking (e.g. when instances are serialized
> through pointers) they are used as indices into a vector so using these
> indices
> keeps overhead to a minimum.  There are cases when an object id is
> assigned but not written to the archive I don't see these as having much
> utility outside of serialization.

So, it's not easily possible to plug a different algorithm? 

>> Ah, I've missed that. Do I need to provide both 'type' and 'value'? Can't
>> serialization library work with just one?
> 
> Could be.  I just provided both so that I could interoperate with mpl
> without having to think about each specific case.

But for user this can be inconvenient.

 >> Actually, my archive initially have only one (non-templated) 'save' for
>> unsigned. I got compile error until I've declared 'save' for const char*.
>> I'm not sure why.
> 
> Its probably because of the above (pointer to a primitive type).  Attempts
> to serialize instances of types whose implementation level is set to
> not-serializable will result in compile time assertions. (these assertions
> are sometimes the reason for the deep nested mpl).

In fact, it looked like the library tried to save char* somewhere.... I'll
take a second look.

>> Right. I think this problem can be addressed with a wrapper for dynamic
>> array
>> 
>>    char* str(0);
>>    ar >> make_dynarray_wrapper(str)
>> 
>> so that the library allocates the string itself.
> 
> How does it know what size to make the array?  Maybe you meen
> 
> Ar >> make_dynarray_wrapper(str, number of elements * element size)

I actually meant.

   int size;
   char* str;

   ar & make_dynarray_wrapper(str, size);

On load, the size is initialized to the size of data and 'str' is new[]-ed.

> Of course for char arrays one could use
> 
> Ar << binary_object(str, element_count * element_size)
> 
> And it would still be portable.

Again, how would I load data? I'd need to new[] the array myself and this
leads to split serialization.

> 
> I have recently got the polymorphic array working on my machine.  Only a
> couple really small changes to the library code were required.  For the
> list of primitive types I included all portable C++ primitive types. (that
> is no long long, __int64 etc.) I'm not sure how much interest the
> polymorphic archive will engender and it's a little hard to understand
> until you've actually spent enough time with the library to appreciate the
> limitations of templated code in certain scenarios. So although its clean
> (and clever) its not going to be easy to explain.

These are great news. It it possible that you make BOOST_EXPORT always
register classes with the polymoprhic archive and use polymoprhic archive
as fallback for serializing classes?

- Volodya