Re: [Boost-users] cross-platfrom binary serialization?

From: troy d. straszheim (troy_at_[hidden])
Date: 2008-08-29 15:03:42

James Sutherland wrote:
> 
> On Aug 29, 2008, at 12:41 PM, troy d. straszheim wrote:
> 
>> But it isn't fast.  If the necessity of bitshuffling makes it 
>> impossible to
>> serialize, say a vector<double> via the optimized array handling, you
>> could easily be talking about a factor of 10 in speed.  Showstopper 
>> for us,
>> at least.   I suppose you could copy the entire buffer, flip all the bits
>> at once, then serialize the flipped buffer, but this also has significant
>> cost, too much for scientific applications where another option exists.
>>
> 
> Have you looked at HDF5?  It supports metadata, parallel IO, arbitrary 
> data structures, etc. and is designed for HPC applications.  It also has 
> native binary format support and will automatically provide 
> cross-platform binary compatibility.

We do use HDF5... With boost.python and pytables plus boost::serializaton
backed c++ datastructures one can quite flexibly provide
converter/extractor/reducer utilities that get you from a boost::serialization
portable binary format to a more 'analysis friendly' hdf5 format.  Works great.

-t