Re: [boost] [serialization] portable_binary_archive

From: troy d. straszheim (troy_at_[hidden])
Date: 2005-09-17 03:22:32

Matthias Troyer wrote:
> 
> In our physics applications we currently use the XDR format with  
> Boost serialization. See 'man xdr' on a Unix system.
> 

Thanks very much for your comments... this is very interesting.   I 
looked back at the list archives and see this xdr archive discussed as 
far back as the serialization code review....  I'd love to see the code. 
  I certainly prefer the notion of xdr over hdf, given that xdr appears 
comparatively lightweight, mature, and it is everywhere:  external 
dependencies cause many of my biggest headaches.   And if it makes this 
portable binary archive thing unnecessary, great.

> This seems to be fine if all you want is compatibility at the level  
> of the least common denominator. By checking if the value of an  
> integer fits into 32 bits you make this library archive useless for  
> people who might need compatibility between 64 bit platforms. 

Right.  The approach is purposefully least common denominator, the 
thinking is that the error should occur as soon as possible.  For 
instance, one could have everything padded out to 64 bits, but it then 
could easily happen (in my world) that somebody writes a terabyte of 
simulation data that can't be read on a 32 bit machine and then throws a 
temper tantrum.  But perhaps idiot-proofing to this degree isn't 
warranted for a boost component, given that it ruins 64bit<->64bit 
portability...

This could easily be made configurable, though: the archive could store 
a list of sizes for each primitive type in the header (as the binary 
archive does already), and the iarchive could upsize/downsize and 
range-check as necessary.  The constructor of an oarchive could take 
size modifiers as arguments:

ostream os;
portable_binary_oarchive(os, size_of<short>(4), size_of<int>(8));

or something like that.   You could put an endianness flag in there as 
well.  One can then force some types to be written with smaller type 
sizes as needed, and if one does nothing, one has compatibility between 
machines with equivalent primitive type sizes.  Not clear to me what a 
good default would be.

While I'm thinking out loud here, it crossed my mind to allow changing 
the save or load sizes of primitive types on an archive that is already 
open with something iomanip-ish, e.g.

my_portable_binary_oarchive << setsize<short>(4);

as maintaining this state would take a prohibitive amount of space, and 
doesn't go with the "you don't pay for what you don't use" philosophy.

 > What do
 > you think about the following idea: the portable binary archive
 > implements serialization for the fixed length integers (int32_t,
 > int64_t, ...) and it is the user's responsibility to only use the 32
 > bit integers if they want portability to all 32 bit platforms. Also
 > watch out that on some platforms even short and int are 64 bit.

It sounds good to me; this is essentially the least common denominator 
approach, no?  I can't help thinking that there should be a way to get 
64 bit integers through there as well, via int64_t, but this is 
frustrating me at the moment, I think I've just been staring at this 
code too long.  I'm wondering whether I wouldn't rather abandon it if 
there is (like this xdr_archive) a good alternative already in use.

Thanks,

-t