Boost logo

Boost :

From: Matthias Troyer (troyer_at_[hidden])
Date: 2003-11-17 03:07:28


Dear Robert, dear Boosters

Unfortunately I currently do not find enough time for an in-depth look
at the serialization proposal since I am too busy at work. Looking
through Robert's new proposal I believe that this could be a good
replacement for the binary serialization scheme we are currently using,
if one additional feature could be added that I proposed already during
the last review: I want to make sure that serialization of large data
sets (I'm talking about up to Gigabyte sized containers on 64-bit
machines here) is efficient. We routinely have to serialize large
vectors or matrices and send it over a fast network, or store it to
disk. Currently the only way to do that is

either i) call serialization of a std::vector, which in turn serializes
each element separately, leading to billions of function calls

or ii) serialize it binary, which will always use the binary
representation of the current machine

Since we want to have portable and fast archives neither option is good
enough for us. The first is too slow in some contexts, and the second
one not portable (I might want to change endianness in
(de)serialization).

I thus propose to

* add, in addition to the serialization functions of the basic data
types another set of functions that serialize contiguous arrays of the
POD types.

* add a default implementation that just calls the serialization
function for each element consecutively. Then only those archives for
which the fast implementation are necessary and/or possible have to
implement the array version.

* implement the serialization of C-arrays, boost::array, std::vector,
boost::multi_array, and the ublas data types with contiguos storage to
use the optimized array serialization functions when called for POD
types.

This will cause marginal compile time and no run time overhead for
users who do not need these features, but will allow the use of the
serialization library in high performance applications.

Best regards,

Matthias


Boost list run by bdawes at acm.org, gregod at cs.rpi.edu, cpdaniel at pacbell.net, john at johnmaddock.co.uk