Re: [boost] Re: Using Serialization for binary marshalling

From: Robert Ramey (ramey_at_[hidden])
Date: 2004-04-18 11:38:59

Matthew Vogt  writes:

> Actually, XDR and CDR are of secondary interest for me; 

For me too, but these have be much requested and finishing these off in a
polished way will be significant contribution to the library for many users.

> I'm interested in
> being able to save to proprietary formats (which often means that the 
> applications involved never got around to specifying a standard 
> format...).  There must be thousands of ad-hoc binary formats in use 
> today.

Hmmm - I'm a little wary of this.  Though I'm not sure I know what you mean.
Some persons interested in the library have hoped it can be used to generate
some specific format.  But that format doesn't accommodated all the
information required to rebuild and arbitrary C++ data structure so in order
to do one ends up coupling the serialization of classes to the archive
format - just exactly what the serialization library is designed to avoid.
Actually, My current thinking is to add a section to the documentation ( I
love my documentation ! Many people have contributed to it by careful
reading and criticism) which suggests a transition path from a proprietary
format to usage of the serialization library.  This transition would be
basically

a) make a program which loads all your "old" data in the "old" way.
b) serialize the structures.

So I'm skeptical of trying to adjust to "old" proprietary formats with the
serialization library.

>> 1) I wonder why you derived ordered_oarchive  from
>> basic_binary_oprimitive<Archive, OStream> 

> I'm using the save_binary / load_binary functions from 
> basic_binary_oprimitive
> and basic_binary_iprimitive.

Maybe we'll consider factoring this out into a standalone function.  We'll
keep and eye on this for now.  In fact, if the salient feature of XDR is
Endian awarness, alignment etc.  I'm wondering if some of the functionality
of your classes shouldn't be moved into your own version of load_binary
thereby making inheriting the native one unnecessary.

> 2) there's some stuff in boost that addresses alignment in a guaranteed?
> Portable manner that may relevant here. Sse #include
> <boost/aligned_storage.hpp> . BTW - the best way to make your code
portable
> without cluttering up with #ifdef etc... is to use more boost stuff - let
> other people clutter up their code with #ifdef all over the place. 

> Yes, but the aligned_storage template helps with platform-specific 
> alignment within the machine.  I don't see how it helps with 
> platform-independent alignment within the content of the archive...

OK - I would like to see that made a little more transparent and better
explained with comments.  It's an important part of the issues being
addressed.

> 3) I'm curious about the override for the saving of vector.  ...

> I need to override the serialization of vector, because the vector must be
> serialized with a known policy to yield a required layout in the archive.
> The fact that this serialization happens (at this point in time) to be
> exactly the same as the default implementation is not relevant - that can
>  be changed at any time, but the CDR, XDR and other binary formats must
> not change.

I'm still not convinced - its seems to me that it shouldn't need to be
overridden for XDR and CDR which is what these classes do.  What about list,
deque, set, etc.

[ Points 4-6 conceded ]

>> 7) I'm just a little queasy about using the term marshalling.  In fact
>> any of the archive implementations could be used for marshalling - not 
>> just a binary one (XML is the favorite flavor of the month).  So I 
>> think it's sort of misleading. 

> Yes, I see what you're saying.  I'll have a think about this - but the
> term 'marshalling' is not particularly prevalent in the code.  Even if 
> the term does have broad application, I think I am using it in the 
> traditional sense.

Your library does marshalling ( as understand the term is usually used ).
My complaint is that its too modest.  Your library does more than that.
When I started this library there was strong usage of the term "persistence"
which lead to the misconception that the library had nothing to do with
"marshalling".  I see serialization as use in a number of things -
persistence, marshalling and who knows what else? (e.g. generating a crc on
the whole data state of the program to detected changes).  That's why I went
to much effort to avoid this characterization of the library.  Your addition
will gain strength from leveraging on this and by fitting in with the
established pattern will be found easier to use.  This will make it more
successful.  Also by following such a pattern it will almost entirely
eliminate the need for special documentation.

> 8) in my native binary archive, I used a template parameter for the stream
> class.  As it stands, I don't test or use the corresponding archive for
wide
> character streams because the WEOF would need special treatment.  I
haven't
> spent any time on this.  Perhaps you've addressed this in some way. 

> No, I was hoping you'd have it all sorted out - I've never used a stream
> specialised with anything but char.  In fact, it doesn't really make sense
> to me, to produce binary output as anything but a byte stream - I was just
> following your lead, really :)

That's basically why I haven't spent any time on it. I made it templated on
the stream class basically motivated by symmetry with the other archives. It
should be finished by addressing the WEOF/EOF issue and using a code convert
facet which turns the wchar_t interface into a byte stream.  This way it
would be useful to programs which are built around wchar_t rather than char.
I suspect more programs in the future will do this.  Its not very hard, but
not of urgent interest. BTW - this might be better reason to derive from
basic_binary_oprimitive.  

Robert Ramey