Re: [boost] [serialization] fast array serialization (10x speedup)

From: Matthias Troyer (troyer_at_[hidden])
Date: 2005-10-11 02:57:04

On Oct 10, 2005, at 6:48 PM, Robert Ramey wrote:
>
> That sounds very good to me. Maybe I spoke too soon.
> I don't see how this would require and changes at all to the  
> serialization
> library.
> I don't see why has_fast_array_serialization has to be part of the
> serialization library.
> Maybe all the code can be included in boost/serialization/ 
> fast_arrary.hpp ?
> This header would be included by all the classes that use it and no  
> others.
>
> [snip]

> If has_fast_array_serialization<Archive,Type> is defined
> boost/serialization/fast_arrary.hpp I'm still OK with it.

That's where it is defined in my proposal.

> The only thing I'm missing here is why the serialization library  
> itself has
> to be
> modified to support all this.  It seems that all this could easily be
> encapulated in
> one (or more) separate optional headers.  This would be the best of  
> all
> possible
> worlds.

There are actually only very few modifications:

1. boost/archive/detail/oserializer.hpp and iserializer.hpp require  
modifications for the serialization of C-arrays of fixed length. In  
my version, the class save_array_type is modified to dispatch to  
save_array when fast_array_serialization is possible. The underlying  
problem here is that oserializer.hpp implements the serialization of  
a type here (the C array!). The optimal solution to this problem  
would be to move the array serialization to a separate header boost/ 
serialization/array.hpp, as is done for all C++ classes.

2. boost/serialization/vector.hpp is also modified to dispatch to  
save_array and load_array where possible. I don't think that this is  
a problem?

3. I had to introduce a new strong typedef in basic_archive.hpp:

BOOST_STRONG_TYPEDEF(std::size_t, container_size_type)
BOOST_CLASS_IMPLEMENTATION(boost::archive::container_size_type,  
primitive_type)

I remember that you suggested in the past that this should be done  
anyways. One reason is that using unsigned int for the size of a  
container, as you do it now will not work on platforms with 32 bit  
int and 64 bit std::size_t : the size of a container can be more than  
2^32. I don't always want to serialize std::size_t as the integer  
chosen by the specific implementation either, since that would again  
not be portable. By introducing a strong typedef, the archive  
implementation can decide how to serialize the size of a container.

The further modifications to the library in

boost/serialization/collections_load_imp.hpp
boost/serialization/collections_save_imp.hpp
boost/serialization/vector.hpp

were to change the collection serialization to use the  
container_size_type.

I don't think that you will object to this.

There is actually another hidden reason for this strong typedef:   
Efficient MPI serialization without need to copy into a buffer  
requires that I can distinguish between special types used to  
describe the data structure (class id, object id, pointers, container  
sizes, ...) and plain data members.

Next I have done a few changes to archive implemenations, the only  
important one of which is:

4. boost/archive/basic_binary_[io]archive.hpp serialize  
container_size_type as an unsigned int as done till now. It might be  
better to bump the file version and serialize them as std::size_t.

All the other changes were to modify the binary archives and the  
polymorphic archive to support fast array serialization. In contrast  
to the above points this is optional. Instead we could provide  
fast_binary_[io]archive and fast_polymporphic_[io]archive, that  
differ from their normal versions just by supporting fast array  
serialization. I could live with this as well, although it makes more  
sense in my opinion to just addd the save_array/load_array features  
to the existing archives.

Of all the points above, I believe that you will not have anything  
against points 3 and 4 since you proposed something similar already  
in the past if I remember correctly.

Issue 2 should also be noncontroversial, and the main discussion  
should thus be on issue 1, on how one can improve the design of [io] 
serializer to move the the implementation of array serialization into  
a separate header.

Matthias