Re: [boost] [serialization] fast array serialization (10x speedup)

From: Matthias Troyer (troyer_at_[hidden])
Date: 2005-10-19 13:13:19

On Oct 19, 2005, at 8:26 AM, Robert Ramey wrote:

> Matthias Troyer wrote:
>
>> On Oct 15, 2005, at 10:33 PM, Robert Ramey wrote:
>>> re on different pages.  you only have to override
>>> serialization of vector etc in only one place - in the
>>> fast_archive_adaptor
>>> class.
>>> Then the behavior is available to any class that the adaptor is
>>> applied to.
>>>
>>
>
> [snip - discussion why serialization of multi_array and similar  
> classes is always intrusive]
>
> I've come to realize that some classes do not provide an iterface  
> sufficient
> to support serialization.  share_ptr has this problem as does  
> boost::any.
> I'm sure there are others for as well.  Its out of my hands.
>
>
>> Since Boost.Serialization support has to be intrusive for these data
>> structures, I believe that the intrusiveness should be kept to a
>> minimum and only one serialization function be provided.
>>
>> If the serialization library documentation
>> tells, e.g. the MTL authors to serialize their arrays by looping over
>> all elements, I will have to, after they implement their version, be
>> intrusive on the MTL library to get direct array serialization in.
>> Better to have them support it directly!
>>
>
> fine - just make your own archive. I'm perfectly happy with this.  The
> documentation can easily be changed so that for the archives included
> with the package the default serialization of arrays is ...

Robert, it seems that maybe I did not make myself clear enough, so  
let me stress the important point:

For classes like multi_array, MTL matrices, Blitz arrays, ublas  
matrices, ... in almost all cases serialization has to be intrusive.  
So the question is, what should we recommend to the authors of this  
library: should we encourage them to use the save_array/load_array  
interface whenever it is possible?

The only sensible answer is yes, because

a) it will give them much faster archives (large matrices are almost  
always stored in binary format)
b) overloading the serialization in my own archive class, as you  
suggest, is impossible, since serialization has to be intrusive for  
these classes.
c) it places all the serialization code of these classes with the  
library, and not inside my archive.
d) it is easily extensible (unlike placing all serialization codes  
using save_array, load_array in my archive classes.

>>> One still has to include the header.  This violates the boost
>>> principle -
>>> don't
>>> pay for what you don't use.   Actually the boost principle would be
>>> - don't
>>> even know about what you don't use - which is better anyway.
>>>
>
>
>> You already violate this principle much more severely in the
>> serialization library. If I do not want object tracking and
>> versioning for a text_oarchive of some objects, the code for tracking
>> and versioning is still included by the serialization library.
>>
>
> The headers are included but the code isn't  instantiated.

Oh, and how about the explicit instantiations in the *.cpp files?

if you are concerned with compile time for the instantiated functions  
thanthe save_array/load_aray code is actually better. For example,  
for the binary archives serializing an array with my version is a  
single call to save_array or load array, while in the default version  
it is a for loop.

> This is not an issue of efficiency.  The instanctiated code is the  
> same
> regardless of where you put it.

Except that for multi_array, Blitz array and similar classes the  
serialization code is intrusive, and I cannot just put an overload  
into my archive class. Providing a save_array/load_array function the  
authors of those libraries can provide efficient serialization  
whenever it makes sense at negligible cost.

>
> So its simple just to make your own version
> of binary archives.  The library supports and encourages that and  
> you don't
> have to change anything in the core to do that.  I'm looking  
> forward to
> seeing the final result.

Actually, the archives are available, but we have to change the  
serialization library:

- we have to add the fast array serialization traits class
- we have to encourage library authors to use save_array/load_array  
wherever possible, and has_fast_array_serialization.hpp thus will  
regularly be included
- we have to provide fast array serialization for the STL contiguous  
array-like classes such as std::vector and std::valarray, as well as  
for C-style arrays

Matthias