Re: [boost] [serialization] fast array serialization (10x speedup)

From: Ian McCulloch (ianmcc_at_[hidden])
Date: 2005-11-22 16:18:47

Peter Dimov wrote:

> David Abrahams wrote:
> 
>> 3. An archive can normally only apply an array optimization to a
>>   particular subset of types.  This subset varies by archive type and
>>   can usually be captured by a type trait such as is_POD or
>>   is_fundamental.
> 
> In the situations that I've needed it, the subset of types that could be
> written in a single operation could not be described by a type trait. A
> type trait can't tell you whether the in-memory representation and the
> on-disk representation of a type are the same. I just enumerated the
> types, guarding the overloads with an appropriate #ifdef on endianness.

I use a mapping that specifies how fundamental types are to be represented
on disk (which is also a function of the archive format), and a function to
do the conversion from the in-memory representation to the on-disk
representation.  Part of this is a trait to indicate if the representations
are identical, which enables memcpy optimizations.  The logic to determine
what the actual in-memory representations are for the host platform is
handled by autoconf macros, but the rest is done by metafunctions.

> I don't understand how an archive could handle an array of arbitrary PODs.

It can't, obviously.  I read that as being an (oversimplified) example of
the basic approach of a trait to control optimizations.  Indeed, an obvious
candidate for (non-portable) optimizations is std::complex<double>, which
isn't POD anyway.

> 
>>   We'd like to encapsulate that choice in a base
>>   class template that allows us to avoid writing complex dispatching
>>   logic in each array-optimized archive.
> 
> I usually just add the equivalent of a save_sequence( A&, X*, unsigned )
> overload for every X that is supported by A.

I gather the idea is to make possible some kind of portable binary archive
that does a dispatch based on the desired on-disk format.  And allow an
easy method for users to specify a type is memcpy()-able (or more
accurately, some way of telling the archive what the in-memory format is so
the archive can then decide if it is memcpy-able to the on-disk format) on
*their particular hardware*, without adding an overload and doing that
logic manually.

Cheers,
Ian