Re: [boost] [serialization] portable_binary_archive

From: troy d. straszheim (troy_at_[hidden])
Date: 2005-09-17 03:33:36

Robert Ramey wrote:
> There might be people interested.  You might upload it to the vault.
> 
> You don't have to write a test program.  You can use all the boost archive 
> test programs with your own archives by following the instructions in the 
> documentation  Reference/Archve Class Reference/ Testing.  The batch file 
> run_archive_test.bat will run all the tests with your archive.

Right.  This testing architecture I've used; the 
portable_binary_(i|o)archive and the polymorphic version pass all the
existing serialization tests, alongside the regular binary, text, and
xml archives (I just got all that done, it's sitting here on disk.)

So that much is good.  But of course the real test of such an archive is
to verify that you can correctly read on platform A what was written on
platform B, and vice-versa.  I've done this testing (probably to the
point of overkill), but again with a different testing scheme.  One
thing inherent to the problem is that you are required to have portable
binary archives checked in to source control.

So there is a directory, $SOMEWHERE/test/portable_archives/, in which
there are several subdirectories, each of which contains a file called
"xml" and one called "pbin".  On platform P, when you run the tests, you
first create and open a directory
$SOMEWHERE/test/portable_archives/<my_hostname>, and you serialize a
bunch of stuff out to both file "xml" (in xml, naturally) and "pbin" (in
portable binary) and close them.  Then you go through all the other
directories under $SOMEWHERE/test/portable_archives, (each of which is
named after some other host, preferably on a different architecture),
reading in from "xml" and "pbin" and comparing the results in memory.
Those should all pass, and at the end, the "pbin" files in all the
subdirectories should have the same checksum.

One would probably want to change this a bit, as the hostname was 
convenient but doesn't tell you anything about the architecture.  One 
might want to manufacture a name for the binary that describes the 
architecture in terms of the primitive type sizes, for instance

E-C-S-I-L-LL-F-D.(pbin|xml)

where E is 0 for big-endian, 1 for little-endian, and C is sizeof(char),
S is sizeof(short), I is sizeof(long) etc.

OR, maybe you really don't think all this is worth fooling with, that's 
cool too, I dunno.

-t

> 
> Robert Ramey
> 
> 
> troy d. straszheim wrote:
> 
>>Hi Robert, boost...
>>
>>I've implemented a portable binary archive, wondering if there is
>>general interest.  I've tested (very thoroughly) on our target
>>platforms, which are 32 and 64 bit linux, and osx on a g5, each
>>platform reads and writes archives written by the others, down to the
>>bit.
>>
>>The archive itself is stored little-endian (as the vast majority of my
>>users are on intel hardware) so big-endian platforms are responsible
>>for byteswapping, though this could be configurable.  The problem
>>types are (unsigned) long int and long double.  For long int, the
>>archive checks
>>to see if it is small enough to fit into 32 bits, if so it stores it
>>in 32 bits, otherwise it throws.  For long double, it just throws, as
>>that's 12 bytes on some platforms, 16 on others, no good way to handle
>>this, and the type is seldom used anyhow.  Float and double are
>>assumed
>>to be ieee754, and are just handled as blocks of 32 or 64 bytes.  So
>>if
>>the write succeeds, the read is guaranteed to.
>>
>>Part of the idea here is that without resorting to widespread use of
>>int32_t and friends, 4-byte-long platforms are always be able to write
>>archives readable on 8-byte-long platforms. On 8-byte-long platforms,
>>you could write archives readable on the smaller machines, but best to
>>choose either int or long long int or a typedef thereof (int32_t or
>>int64_t).
>>
>>The code is at http://svn.resophonic.com/pub/boost, in
>>
>>boost/archive/detail/portable_binary_archive.hpp
>>boost/archive/portable_binary_iarchive.hpp
>>boost/archive/portable_binary_oarchive.hpp
>>libs/serialization/src/portable_binary_iarchive.cpp
>>libs/serialization/src/portable_binary_oarchive.cpp
>>libs/serialization/test/portable_binary_archive.cpp
>>
>>Issues that come to mind:
>>The test isn't boost-test, that'd need rewriting.  Haven't tried any
>>of
>>this on microsoft.  Maybe there is some preferred way to detect
>>endianness.  The decision to go little-endian was relatively
>>arbitrary,
>>maybe the byte-swapping should be configurable.    A possibility I
>>considered was to upgrade all 32 bit longs to 64 bit and do the
>>range-check/throw when *reading* (We nixed this idea, preferring to
>>have
>>the error come up sooner rather than later), maybe this should be
>>configurable as well.   There's surely platforms out there with other
>>differences that aren't handled, I don't have a good feel for whether
>>they would make the archive useless to a wider audience.  The tests
>>require having access to binary archives written on other platforms,
>>haven't yet thought about how to put that together in the boost
>>build/test context.
>>
>>So have a look, let me know if you're interested.  Thanks again for a
>>great lib.
>>
>>troy d. straszheim
>>
>>
>>
>>
>>
>>
>>_______________________________________________
>>Unsubscribe & other changes:
>>http://lists.boost.org/mailman/listinfo.cgi/boost 
> 
> 
> 
> 
> _______________________________________________
> Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
>