Re: [Boost-users] cross-platfrom binary serialization?

From: troy d. straszheim (troy_at_[hidden])
Date: 2008-08-29 14:41:44

Rainer Deyke wrote:
> Johan Råde wrote:
>> I see three options for dealing with the endianness issue:
>> 1. make all archives big-endian
>> 2. make all archives little-endian
>> 3. use the native format when saving, and put an endianness flag in 
>> the archive.
>>
>> 1 is inefficient when moving data between little-endian platforms
>> 2 is inefficient when moving data between big-endian platforms
>> So 3 should be most efficient. Is there an easy way of storing an 
>> endianness-flag in an archive?
> 
> I don't think performance should be the overriding concern, especially 
> since byte-shuffling is very fast.  

But it isn't fast.  If the necessity of bitshuffling makes it impossible to
serialize, say a vector<double> via the optimized array handling, you
could easily be talking about a factor of 10 in speed.  Showstopper for us,
at least.   I suppose you could copy the entire buffer, flip all the bits
at once, then serialize the flipped buffer, but this also has significant
cost, too much for scientific applications where another option exists.

The problem with option 3 is that it introduces a potential source of
 > bugs that only manifests when moving
> between platforms with different endianness.  I'd prefer option 1, 
> precisely because it requires shuffling on the most common platforms so 
> any bugs in the shuffling code are sure to be caught early.

Actually this is very easy to test for, even if you don't have machines of the
other endianness available.   (the md5sum of the generated archive must match
for all platforms, and these sums can be checked in to svn)

-t