Re: [Boost-users] cross-platfrom binary serialization?

From: James Sutherland (James.Sutherland_at_[hidden])
Date: 2008-08-29 14:50:37

On Aug 29, 2008, at 12:41 PM, troy d. straszheim wrote:

> Rainer Deyke wrote:
>> Johan Råde wrote:
>>> I see three options for dealing with the endianness issue:
>>> 1. make all archives big-endian
>>> 2. make all archives little-endian
>>> 3. use the native format when saving, and put an endianness flag  
>>> in the archive.
>>>
>>> 1 is inefficient when moving data between little-endian platforms
>>> 2 is inefficient when moving data between big-endian platforms
>>> So 3 should be most efficient. Is there an easy way of storing an  
>>> endianness-flag in an archive?
>> I don't think performance should be the overriding concern,  
>> especially since byte-shuffling is very fast.
>
> But it isn't fast.  If the necessity of bitshuffling makes it  
> impossible to
> serialize, say a vector<double> via the optimized array handling, you
> could easily be talking about a factor of 10 in speed.  Showstopper  
> for us,
> at least.   I suppose you could copy the entire buffer, flip all the  
> bits
> at once, then serialize the flipped buffer, but this also has  
> significant
> cost, too much for scientific applications where another option  
> exists.
>

Have you looked at HDF5?  It supports metadata, parallel IO, arbitrary  
data structures, etc. and is designed for HPC applications.  It also  
has native binary format support and will automatically provide cross- 
platform binary compatibility.

It will likely require a bit more code instrumentation than boost  
serialization, but may be worth it if performance is key...