[boost] Re: Using Serialization for binary marshalling

From: Matthew Vogt (mvogt_at_[hidden])
Date: 2004-04-19 18:43:56

Robert Ramey <ramey <at> rrsd.com> writes:

> > Well, I never thought these were necessary, remembering that I am
> > providing for classes, which are designed to be serialized into a
> > particular binary format.  In a binary format, all collections will boil
> > down to a group of repetitions, which are either preceded by a
> >  length/count argument, or whose length is a defined property of the 
> > format itself.
> 
> Correct. That's exactly what the current implementations of STL classes do.
> So my question is why override vector for you xdr/cdr archives?

Primarily, because the implementation you have chosen for default serialization
of STL containers is just an implementation detail from your point of view;
you could quite reasonably change it in the next version, if someone with 
alternate requirements requested that, and it didn't conflict with your own
purposes.  OTOH, XDR and CDR mandate how sequences should appear in a
stream, so it is necessary to ensure that the default implementation is not
relied upon for conformance.

Another reason, is that other formats will vary the implementation in minor
ways - by using a preceding length field in 8-bit signed format, for example.

> > For me, vector has always sufficed, in either fixed_length<>, or 
> > variable_length<> guise.  Perhaps others have differing experience.
> 
> For your xdr/cdr archives to appealing to the boost users that have asked
> for this and for these archives to be tested to the same standards, all STL
> collections will have to be handled.  

Ok.  I'll add this, although I would be mildly surprised to see it in use.

> > But there are inherent limitations in marshalling (per my usage). 
> > Pointers, which are adeptly handled by the general library, are not valid
> > elements of a marshalled data set.
> 
> I disagree here, I think the ability to pass "equivalent" pointers across
> space( marshalling ) is just as valuable as the ability to pass them acrose
> time ( persistence ). 

Yes, but you're describing a higher-level view of marshalling. Such protocols
certainly exist, and may well be desirable for boost users.  They are, however,
implemented in terms of CDR, XDR or equivalent, so I may as well get these
right first.

> This puts a fine point on my views.  I think that in the past, much of the
> difficulty of the topic was due the fact that different people viewed in
> terms of marshalling and other in terms of persistence and there is a lot of
> overlap.  (One attempt to define "serialization" when to 200 words !).  I
> believe my definition as stated in the start of the documentation factored
> out the commonality and I resolved to focus on that.  I also believe that
> this has turned out to be one of the main reasons that the library has been
> embraced to the extent that it has.
> 
> This explains my reluctance to consider Marshalling as a special case.

I think it is appropriate to view marshalling as a degenerate case, utilising
the serialization framework, but not fulfilling its functional range.

> I would like to see your xdr/cdr archives finished off at the same level of
> generality as the other archives.  Any special purpose marshalling would
> leverage off that.

I don' think this is possible, since these formats do not specify sufficient
types to do so.  It would be possible to allow type promotion to cover
some missing types - for example, in XDR, it would be possible to allow all
C++ integral types less than 32-bits to be promoted to 32-bits during
serialization, but I think this would be misleading.  

More problematically, there is no concept of objects in this situation. It
wouldn't make sense to transmit objects via pointers, since 'the other side'
will never reconstitute objects.  Eliding extraneous copies of already-
serialized objects would be an error.

Matt