Re:[boost] Serialization of large data sets

From: Robert Ramey (ramey_at_[hidden])
Date: 2003-11-17 11:14:31

Matthias Troyer wrote:

>I thus propose to

>* add, in addition to the serialization functions of the basic data 
>types another set of functions that serialize contiguous arrays of the 
>POD types.

>* add a default implementation that just calls the serialization 
>function for each element consecutively. Then only those archives for 
>which the fast implementation are necessary  and/or possible have to 
>implement the array version.

The current implemenation already has this.

>* implement the serialization of C-arrays, boost::array, std::vector, 
>boost::multi_array, and the ublas data types with contiguos storage to 
>use the optimized array serialization functions when called for POD 
>types.

I believe all this can be addressed by implementing specifice
load/save_override functions in an archive for specific datatypes and/or
templates.  I am currently improving the implementation to make it easier 
to derive one's own custom archive from one of the provided ones.  
Presumably such this would be used to provide special/different
handling for situations as you describe above.  The next upload
draft will include these improvements along with a case study
showing how to handle cases such as the above.

Unfortunately, using this facility will require a careful review
of most of the manual.

BTW - currently it would be easy to make a custom version 
of binary_i/oarchive which is the simplest of the lot.

Robert Ramey