From: troy d. straszheim (troy_at_[hidden])
Date: 2005-09-17 03:22:32
Matthias Troyer wrote:
> In our physics applications we currently use the XDR format with
> Boost serialization. See 'man xdr' on a Unix system.
Thanks very much for your comments... this is very interesting. I
looked back at the list archives and see this xdr archive discussed as
far back as the serialization code review.... I'd love to see the code.
I certainly prefer the notion of xdr over hdf, given that xdr appears
comparatively lightweight, mature, and it is everywhere: external
dependencies cause many of my biggest headaches. And if it makes this
portable binary archive thing unnecessary, great.
> This seems to be fine if all you want is compatibility at the level
> of the least common denominator. By checking if the value of an
> integer fits into 32 bits you make this library archive useless for
> people who might need compatibility between 64 bit platforms.
Right. The approach is purposefully least common denominator, the
thinking is that the error should occur as soon as possible. For
instance, one could have everything padded out to 64 bits, but it then
could easily happen (in my world) that somebody writes a terabyte of
simulation data that can't be read on a 32 bit machine and then throws a
temper tantrum. But perhaps idiot-proofing to this degree isn't
warranted for a boost component, given that it ruins 64bit<->64bit
This could easily be made configurable, though: the archive could store
a list of sizes for each primitive type in the header (as the binary
archive does already), and the iarchive could upsize/downsize and
range-check as necessary. The constructor of an oarchive could take
size modifiers as arguments:
portable_binary_oarchive(os, size_of<short>(4), size_of<int>(8));
or something like that. You could put an endianness flag in there as
well. One can then force some types to be written with smaller type
sizes as needed, and if one does nothing, one has compatibility between
machines with equivalent primitive type sizes. Not clear to me what a
good default would be.
While I'm thinking out loud here, it crossed my mind to allow changing
the save or load sizes of primitive types on an archive that is already
open with something iomanip-ish, e.g.
my_portable_binary_oarchive << setsize<short>(4);
as maintaining this state would take a prohibitive amount of space, and
doesn't go with the "you don't pay for what you don't use" philosophy.
> What do
> you think about the following idea: the portable binary archive
> implements serialization for the fixed length integers (int32_t,
> int64_t, ...) and it is the user's responsibility to only use the 32
> bit integers if they want portability to all 32 bit platforms. Also
> watch out that on some platforms even short and int are 64 bit.
It sounds good to me; this is essentially the least common denominator
approach, no? I can't help thinking that there should be a way to get
64 bit integers through there as well, via int64_t, but this is
frustrating me at the moment, I think I've just been staring at this
code too long. I'm wondering whether I wouldn't rather abandon it if
there is (like this xdr_archive) a good alternative already in use.
Boost list run by bdawes at acm.org, gregod at cs.rpi.edu, cpdaniel at pacbell.net, john at johnmaddock.co.uk