Re: [boost] Re: Using Serialization for binary marshalling

From: Jeff Garland (jeff_at_[hidden])
Date: 2004-04-18 13:43:23

On Sun, 18 Apr 2004 09:38:59 -0700, Robert Ramey wrote
> Matthew Vogt  writes:
> 
> > Actually, XDR and CDR are of secondary interest for me; 
> 
> For me too, but these have be much requested and finishing these off 
> in a polished way will be significant contribution to the library 
> for many users.
> 
> > I'm interested in
> > being able to save to proprietary formats (which often means that the 
> > applications involved never got around to specifying a standard 
> > format...).  There must be thousands of ad-hoc binary formats in use 
> > today.
> 
> Hmmm - I'm a little wary of this.  Though I'm not sure I know what 
> you mean. Some persons interested in the library have hoped it can 
> be used to generate some specific format.  But that format doesn't 
> accommodated all the information required to rebuild and arbitrary 
> C++ data structure so in order to do one ends up coupling the 
> serialization of classes to the archive format - just exactly what 
> the serialization library is designed to avoid. Actually, My current 
> thinking is to add a section to the documentation ( I love my 
> documentation ! Many people have contributed to it by careful 
> reading and criticism) which suggests a transition path from a proprietary
> format to usage of the serialization library.  This transition would 
> be basically
> 
> a) make a program which loads all your "old" data in the "old" way.
> b) serialize the structures.
> 
> So I'm skeptical of trying to adjust to "old" proprietary formats 
> with the serialization library.

Well, I believe you are taking too narrow a view here.  These 'old'
proprietary formats may not be old or replaceable.  They are just part of the
landscape on many projects.  For example, I recently worked on a project where
there was a communications protocol between a server and remote computers that
are on the other end of radio link.  The communications protocols between the
server and the remote computers was a binary format that was rather limited
and specific -- about 4 message types.  The physical level protocol isn't a
standard TCP or UDP b/c the hardware doesn't support this sort of protocol
(for good reason). There are little details like a maximum message size which
means that continuation packets need to be generated. When messages come to
the server from the remote computers it has to modify the message slightly
then transmit the data to other servers and clients over TCP.  Bottom line --
it was very handy to have a different message format on the TCP side as
compared to the proprietary side.

Of course in the servers and clients we had c++ objects to represent the
messages.  Trivially we created classes that had data members for each field
in the message protocol. We had a serialization framework and we wrote
serialization code for each message type.  We serialized the fields in the
order that the protocol specified.  Then we had 2 archive types -- one to
read/write the binary protocol format and one to read/write the internal format. 

This design works great.  There is a clear mapping from the protocol
specification to the message classes and all the details of the 'physical'
protocol are in a single archive class.  Of course if you sent some sort of
message down pipe that wasn't of the correct type to go to the remote
computers, the serialization part of the framework won't stop you.  But we
prevented that in other layers of the architecture.

In this case, our server to client format could easily use one of the standard
archives provided by the library.  But the external protocol could not --
hence the need for the ability to write custom archive types that read/write
proprietary protocols.

As for the design coupling issue, there really isn't a problem.  The archive
implementation is tied to the serialization library as are the serialized
classes.  The fact that an arbitrary c++ type won't work with the specialized
archive isn't important because other elements of the design can easily
prevent this from happening.

I believe that Matthew has now demonstrated that archive extension is
possible, which I believe is one of the major changes from the first review. 
  That said, any documentation of this process would clearly be a plus.  In a
typical proprietary archives writing of the version and type numbers will need
 to be shut off.  So I can imagine documenting these sorts of issues.

Jeff