Re: [boost] [serialization] fast array serialization (10x speedup)

From: Peter Dimov (pdimov_at_[hidden])
Date: 2005-11-22 15:08:56

Ian McCulloch wrote:
> Peter Dimov wrote:
>
>> template<class Archive, class It>
>> inline void save_sequence(Archive & ar, It it, unsigned count)
>> {
>>     while(count-- > 0){
>>         //if(0 == (ar.get_flags() &
>>                 boost::archive::no_object_creation)) // note borland
>>                 emits a no-op without the explicit namespace
>>                 boost::serialization::save_construct_data_adl(ar,
>> &(*it), 0U);
>>         ar << boost::serialization::make_nvp("item", *it++);
>>     }
>> }
>>
>> template<class Archive, class Container>
>> inline void save_collection(Archive & ar, const Container &s)
>> {
>>     // record number of elements
>>     unsigned int count = s.size();
>>     ar << make_nvp("count", const_cast<const unsigned int &>(count));
>>     save_sequence( ar, s.begin(), count );
>> }
>>
>> unless I'm missing something fundamental.
>>
>> So what's all the fuss about?
>
> That isn't quite all that needs to be done.

You are right, a std::vector needs to be special-cased to use a pointer.

> (1) minor nit: an interface that uses (iterator, size) would be
> better than
> a container-based algorithm because that would make it easier to do
> optimizations based on the iterator type (eg, memcpy, or MPI
> operations in the case of a pointer, or maybe some kind of
> distributed iterator in combination with a parallel IO library?).

I don't understand.

>> template<class Archive, class It>
>> inline void save_sequence(Archive & ar, It it, unsigned count)

looks decidedly (iterator, size) based to me.

> Also, the collection isn't necessarily in the form of a container
> (although a proxy container would probably suffice for that case, and
> come to think of it, to handle resizing the container on load it
> might actually be preferable).

I don't understand this either.

> (2) another minor nit: it is probably more convenient to handle the
> details of save_sequence() inside the archive (similarly to other
> primitive types), rather than as a free function.

The point is that you can overload the free function inside your archive's 
namespace.

> (3) : save_collection() [or some functional equivalent] isn't part of
> the public interface of the serialization library.  For whatever
> reason this seems to be the sticking point.

save_collection isn't - and probably shouldn't - but save_sequence would be.