Re: [boost] [serialization] fast array serialization (10x speedup)

From: David Abrahams (dave_at_[hidden])
Date: 2005-11-24 12:15:51

Matthias Troyer <troyer_at_[hidden]> writes:

> On Nov 24, 2005, at 5:24 PM, Peter Dimov wrote:
>
>> David Abrahams wrote:
>>
>>> To be fair, I haven't done the analysis: are you sure your approach
>>> doesn't lead to an MxN problem (for M archives and N types that need
>>> to be serialized)?
>>
>> Yes, it does, in theory. Reality isn't that bad. For every M, the
>> archive author has already added the necessary overloads for every
>> "fundamental" type that supports optimized array operations. 

In most cases this shouldn't require a separate overload for every
fundamental type, since the actual array serialization procedure is
often the same for _every_ type that can use the optimization.  But
again, that's a question of how much work the archive author is
required to do, so Robert may not consider it a valid argument.

>> This leaves a number of user-defined types n (because the number is
>> smaller than N), times M.
>>
>> In addition, even if the author of an UDT hasn't provided an
>> overload for a particular archive A, the user can add it
>> himself. The m*n number for a particular codebase is bounded, and
>> the overloads are typically one- liners.
>
> What if the number n is infinite (e.g. all possible structs
> consisting only of fundamental types), which is what Robert calls
> "bitwise serializable"?

No, we can't detect that category automatically today.

However, our approach is designed to make it very easy and more
foolproof to provide the necessary information for such a type: in
Matthias' original design, a trait specialization; in the more recent
design, an overload of optimize_array that returns mpl::true_.  In
your proposal you need a separate overload of save_sequence.  Then you
likely either have to duplicate the normal fast serialization
procedure in your overload or you have to dispatch to something
written by the archive author that may be named differently for every
archive.  This work is not (necessarily) done by the archive author --
it may be done by the author of a type that needs to be serialized, or
a 3rd party user.  We prefer to add a small amount of additional
framework to avoid that problem.

Furthermore, Doug Gregor has designed and implemented (in GCC) a core
language extension idea we had in Mont Tremblant that allows us to
enumerate all the members of a class.  We plan to propose that for
standardization.  We'd like to see a design that can immediately take
advantage of such a feature when/if it becomes available.

-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com