[boost] Serialization of large data sets

From: Matthias Troyer (troyer_at_[hidden])
Date: 2003-11-17 03:07:28

Dear Robert, dear Boosters

Unfortunately I currently do not find enough time for an in-depth look 
at the serialization proposal since I am too busy at work. Looking 
through Robert's new proposal I believe that this could be a good 
replacement for the binary serialization scheme we are currently using, 
if one additional feature could be added that I proposed already during 
the last review: I want to make sure that serialization of large data 
sets (I'm talking about up to Gigabyte sized containers on 64-bit 
machines here) is efficient. We routinely have to serialize large 
vectors or matrices and send it over a fast network, or store it to 
disk. Currently the only way to do that is

either i) call serialization of a std::vector, which in turn serializes 
each element separately, leading to billions of function calls

or ii) serialize it binary, which will always use the binary 
representation of the current machine

Since we want to have portable and fast archives neither option is good 
enough for us. The first is too slow in some contexts, and the second 
one not portable (I might want to change endianness in 
(de)serialization).

I thus propose to

* add, in addition to the serialization functions of the basic data 
types another set of functions that serialize contiguous arrays of the 
POD types.

* add a default implementation that just calls the serialization 
function for each element consecutively. Then only those archives for 
which the fast implementation are necessary  and/or possible have to 
implement the array version.

* implement the serialization of C-arrays, boost::array, std::vector, 
boost::multi_array, and the ublas data types with contiguos storage to 
use the optimized array serialization functions when called for POD 
types.

This will cause marginal compile time and no run time overhead for 
users who do not need these features, but will allow the use of the 
serialization library in high performance applications.

Best regards,

Matthias