Re: [boost] [serialization] fast array serialization (10x speedup)

From: Matthias Troyer (troyer_at_[hidden])
Date: 2005-11-23 09:09:17

On Nov 23, 2005, at 2:31 PM, Peter Dimov wrote:

> David Abrahams wrote:
>> "Peter Dimov" <pdimov_at_[hidden]> writes:
>>
>>> Conceptual issues with (2) aside (the external format of X is
>>> determined by X itself and you have no idea whether the structure of
>>> X[0] also describes X[1]),
>>
>> I don't know what you mean.
>
> When your mpi_archive is given an arbitrary X, there is no way to know
> whether serializing an array of X can be transformed into  
> serializing a X
> and then building an array datatype out of the result. That's  
> because X's
> serialize functions can do _anything_. Your archive is simply not  
> allowed to
> alter them. (*)

That's why this will not work for all types. It will however work for  
those types which you are most likely to serialize in huge quantities  
in a numerical simulation: real, complex and integer numbers, as well  
as simple structs built from them.

> That is why the optimization must be explicitly enabled by the  
> author of X.
> My approach deals with that by allowing him to write a save_sequence
> overload for X:
>
> void save_sequence( mpi_archive & a, X const * x, size_t n )
> {
>     a.save_pod_array( x, n );
> }
>
> You are trying to generalize this by allowing X's author to define  
> a trait
> that isn't archive specific.

I don't think that Dave has ever claimed that. In my proposal I had  
defined a traits class

has_fast_array_serialization<Archive,X>

which depends on both and can be fully or partially specialized by  
the author of X. However, partial specialization ois not supported by  
all compilers.

Dave's approach is to instead use a meta function inside the archive,  
which avoids the partial specialization problem. In case of a non- 
portable binary archive that meta function might use a traits class  
like Robert's is_bitwise_serializable<X> which the user of X can  
specialize. In the case of the MPI archive, there could be a trait  
like is_mpi_datatype<X> which tells the library whether the  
optimization can be used. In other archives, there might only be a  
fixed set of types just as in your proposal.

Also, Dave's proposal has a free function such as

template <class Archive, class X>
save_array(Archive& ar, X const * n, std::size_t n)

which you could overload as well, just like in your proposal

Matthias