Re:[boost] Re:Re:Formal review: serialization

From: Robert Ramey (ramey_at_[hidden])
Date: 2004-04-22 11:53:30

Vladimir Prus wrote:
> 
> Robert Ramey wrote:
> 

> I guess I have two questions:
> 1. Won't serialization fail in some way if I just seek the stream to the
> position found in index and try reading.
> 2. For random access I need to make sure that all saved objects have
> export
> key. How do I do that? Not necessary out-of-the box but where I can plug
> the check?
> 

I believe the original question referred the ability to "browse" the archive
for purposes of debugging etc.  My view would be that the best way to do
this would be to create and xml_archve.  Your use case suggested a 30 data
would probably would end up as a 90MB XML datafile.  The reason for thinking
in XML would be that I would hope that here exists "XML Browser for large
files" which would be suitable for this case.  If such a thing doesn't exist
- I would prefer spending time making this general purpose tool - (and maybe
a commercial product) to making a custom archive of narrow purpose.

Random access into an archive would require some thought.  First, it's not
clear how it would be used.  In general archives are nested structures, one
could de-serialize an inner piece - but to where? The original data
structure was embedded in something else which is not now there.  So this
can really only consider in the context of a specific application.  Here are
a couple of plausible scenarios.

a) A general purpose archive browser - this could browse the archive in a
random manner but wouldn't actually de-serialize the data.  I don't see any
real problem here.  One would need to create an index either as a side
effect of archive create or with a second pass over the final archive.

b) using serialization for a data-state logging.

Log(Archive ar, statedata){
        // save seek point to index
        // append to archive
        ar << statedata;
}

Recover(Archive ar, statedata &, seekpoint)
        // set stream seekpoint
        ar >> statedata
}

I could envision something like this being made to work.

So I would say that generally, serialization would be expected to be a
serial operation. (surprise!).   On the other hand, in certain special
situations it might be possible/convenient to provide for some random access
but I would expect that to be application specific.

> >> The reason why I think the other is important, it that's it's actually
> >> saving/loading support for plain C++ array -- which is rather basic
> >> thing.
> >>
> >
> > Hmmm - the library already implements serialization of plane C++ arrays
> by
> > serializing each element.
> 
> For *fixed-size* arrays. But not for dynamically allocated arrays. BTW, it
> seems we need two wrappers for completeness: 

> one for dynamic arrays with  element-wise save 

I would envision one using

ar << element_count
For(i = 0; I < element_count; i++)
        ar << element[i]

I'm not sure that its worth adding and documenting such an obvious and
trivial thing as a separate wrapper.

> and another for dynamic arrays with binary save.

I see the binary object as filling that role.

Ar << make_binary_object(dynamic_array_address, element_size *
element_count);

I once did propose a manual section name "Serialization Wrappers" which
would have a couple of examples and highlight the fact that nvp and
binary_object are instances of this concept.  The idea was received
unenthusiastically at the time but now I'm becoming convinced It's more
effective to explain and illustrate the concept rather than try to
anticipate all the wrappers that users might need.  Actually, I think having
such a section would have avoided the confusion surrounding the intended
usage and purpose of binary_object.

> I agree. I actually have a *crazy* but cute idea that one can use file
> offset for object id. How object ids are assigned and can I customize that
> process? That would keep overhead at absolute minimum.

Object_id are assigned sequentially starting with 0.  They are only used for
classes which require tracking (e.g. when instances are serialized through
pointers) they are used as indices into a vector so using these indices
keeps overhead to a minimum.  There are cases when an object id is assigned
but not written to the archive I don't see these as having much utility
outside of serialization.

> Ah, I've missed that. Do I need to provide both 'type' and 'value'? Can't
> serialization library work with just one?

Could be.  I just provided both so that I could interoperate with mpl
without having to think about each specific case.

> >> It also necessary to provide overload for char*. Does not count as
> >> primitive
> >> type?
> >
> > Default implementation for char * will work as it does for other
> pointers.
> 
> Eh... I though that serialization of pointers to builtin types is just not
> allowed.
> 

Default implementation level for pointers to primitives is set to
not-serializable.  This is resetable as it is for any other type.  I doubt
that on would want to do it for something like int or double as I would
expect a lot of unintended side effects.  One might want to use
STRONG_TYPEDEF to define another int or double type and reset the
implementation level of that - but that going off on a tangent.

> Actually, my archive initially have only one (non-templated) 'save' for
> unsigned. I got compile error until I've declared 'save' for const char*.
> I'm not sure why.

Its probably because of the above (pointer to a primitive type).  Attempts
to serialize instances of types whose implementation level is set to
not-serializable will result in compile time assertions. (these assertions
are sometimes the reason for the deep nested mpl).

> > I came to conclude it presented a big security risk.  The problem is the
> > following:
> 
> > later:
> >
> > char str[MAX_STRING_SIZE]
> >
> > ar >> static_cast<char *>(str); // to avoid str being treated as an
> array
> >
> > suppose the text archive gets corrupted to:
> >
> > 3000 abc............
> >
> > The archive will load with a buffer overrun - a security risk.
> 
> Right. I think this problem can be addressed with a wrapper for dynamic
> array
> 
>    char* str(0);
>    ar >> make_dynarray_wrapper(str)
> 
> so that the library allocates the string itself.

How does it know what size to make the array?  Maybe you meen

Ar >> make_dynarray_wrapper(str, number of elements * element size)

Of course for char arrays one could use

Ar << binary_object(str, element_count * element_size)

And it would still be portable.

> >> Can I define only one 'load' for unsigned int?
> >
> > As oppose to ?
> 
> As opposed to overloads for all builtin types. For polymorphic_archive we
> can't have templated function which 'fallbacks' anywhere. We need to have
> a
> closed set of virtual functions and I wonder what's the minimal set.
> > the rest.  BTW this provided a huge benefit. In the original version of
> > last year I got into a never-ending battle to specify virtual functions
> > which was
> > dependent on the compiler - long long, etc.  It was hopeless - moving to
> > templates solved that.
> 
> :-( I guess we're back to those problems.
> 

I have recently got the polymorphic array working on my machine.  Only a
couple really small changes to the library code were required.  For the list
of primitive types I included all portable C++ primitive types. (that is no
long long, __int64 etc.) I'm not sure how much interest the polymorphic
archive will engender and it's a little hard to understand until you've
actually spent enough time with the library to appreciate the limitations of
templated code in certain scenarios. So although its clean (and clever) its
not going to be easy to explain.

Robert Ramey