Re: [boost] [serialization] fast array serialization (10x speedup)

From: Ian McCulloch (ianmcc_at_[hidden])
Date: 2005-11-22 14:17:11

Peter Dimov wrote:

> David Abrahams wrote:
> 
>>  We want to be able to capitalize on the existence of those APIs, and
>>  to do that we need a "hook" that will be used whenever a contiguous
>>  sequence is going to be (de)serialized.  No such hook exists in
>>  Boost.Serialization.
> 

[snip]

> Looking at collections_save_imp.hpp:
> 
> template<class Archive, class Container>
> inline void save_collection(Archive & ar, const Container &s)

[...]

> 
> all that needs to be done is:
> 
> template<class Archive, class It>
> inline void save_sequence(Archive & ar, It it, unsigned count)
> {
>     while(count-- > 0){
>         //if(0 == (ar.get_flags() & boost::archive::no_object_creation))
>                 // note borland emits a no-op without the explicit
>                 namespace
>                 boost::serialization::save_construct_data_adl(ar, &(*it),
> 0U);
>         ar << boost::serialization::make_nvp("item", *it++);
>     }
> }
> 
> template<class Archive, class Container>
> inline void save_collection(Archive & ar, const Container &s)
> {
>     // record number of elements
>     unsigned int count = s.size();
>     ar << make_nvp("count", const_cast<const unsigned int &>(count));
>     save_sequence( ar, s.begin(), count );
> }
> 
> unless I'm missing something fundamental.
> 
> So what's all the fuss about?

That isn't quite all that needs to be done.

(1) minor nit: an interface that uses (iterator, size) would be better than
a container-based algorithm because that would make it easier to do
optimizations based on the iterator type (eg, memcpy, or MPI operations in
the case of a pointer, or maybe some kind of distributed iterator in
combination with a parallel IO library?).  Also, the collection isn't
necessarily in the form of a container (although a proxy container would
probably suffice for that case, and come to think of it, to handle resizing
the container on load it might actually be preferable).

(2) another minor nit: it is probably more convenient to handle the details
of save_sequence() inside the archive (similarly to other primitive types),
rather than as a free function.

(3) : save_collection() [or some functional equivalent] isn't part of the
public interface of the serialization library.  For whatever reason this
seems to be the sticking point.  Making it an optional add-on is OK, but
you really want people to use it _by default_, otherwise you need to go and
rewrite all their serialization functions to make use of whatever
additional functionality the archive provides.

Cheers,
Ian