Re: [boost] [serialization] fast array serialization (10x speedup)

From: David Abrahams (dave_at_[hidden])
Date: 2005-11-22 19:03:59

Ian McCulloch <ianmcc_at_[hidden]> writes:

> Peter Dimov wrote:
>
>> David Abrahams wrote:
>> 
>>> 3. An archive can normally only apply an array optimization to a
>>>   particular subset of types.  This subset varies by archive type and
>>>   can usually be captured by a type trait such as is_POD or
>>>   is_fundamental.
>> 
>> In the situations that I've needed it, the subset of types that
>> could be written in a single operation could not be described by a
>> type trait. A type trait can't tell you whether the in-memory
>> representation and the on-disk representation of a type are the
>> same. 

As soon as you talk about "on-disk" you're already limiting your
thinking too much, since we're not necessarily serializing to disk,
and the idea of having the "same representation" somewhere is somewhat
limiting too.  An MPI archive builds up a skeletal representation of
what needs to be serialized and then MPI reads it out of memory into
the hardware.

For example, MPI archives support array serialization for all PODs
that are not pointers and do not contain pointer members.  Other
archives have different requirements (I'm sorry that I don't remember
what they are but Matthias can give some examples).  You can detect a
subset of the array-serializable types for any given archive with a
type trait, and allow specializations to get optimization for other
types.

>> I don't understand how an archive could handle an array of arbitrary PODs.
>
> It can't, obviously.  

Well, technically it depends how pointers need to be treated.  It may
be that nested pointers are serialized in a separate pass.

> I read that as being an (oversimplified) example of the basic
> approach of a trait to control optimizations.

Right.

> Indeed, an obvious candidate for (non-portable) optimizations is
> std::complex<double>, which isn't POD anyway.

Yes that's true.  Actually POD-ness may be less important than whether
the type has a trivial destructor.

>>>   We'd like to encapsulate that choice in a base class template
>>>   that allows us to avoid writing complex dispatching logic in
>>>   each array-optimized archive.
>> 
>> I usually just add the equivalent of a save_sequence( A&, X*, unsigned )
>> overload for every X that is supported by A.
>
> I gather the idea is to make possible some kind of portable binary
> archive that does a dispatch based on the desired on-disk format.

No, we keep telling everyone: this is not about portable binary
archives (although they will probably benefit), and it's not about
on-disk anything.  The idea is to take advantage of APIs that can
quickly serialize/deserialize contiguous sequences of T, for any T.

-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com