Re: [boost] Serialization Library Review

From: Matthias Troyer (troyer_at_[hidden])
Date: 2002-11-17 14:53:28

On Sunday, November 17, 2002, at 07:08 PM, Robert Ramey wrote:

> Date: Sun, 17 Nov 2002 10:19:23 +0100
> From: Matthias Troyer <troyer_at_[hidden]>
> It is mentioned in several places in the code, docs and in this list
> that the native binary archive derivations have absolutly no
> pretentensions to portabability.  ABSOLUTELY NONE.  Knowing that
> many users will ignore this admonishment, I included code
> do double check that at least this size and endian ness of
> the data types match between the loading and saving machine
> so that the most obvious blunders will be detected and an exception be
> thrown. I don't know what else I can do. The current
> library has NO instance of a portable binary archive.  Many have
> written to tell me how easy it is but I'm still waiting for someone
> to submit one.

I believe we all agree that portable binary archive formats are 
essential in addition to the text based one.
This discussion started because I asked you, why you don't consider 
using int8_t, int16_t, int32_t in addition to int64_t as the primitive 
types? That would make implementing a portable binary archive MUCH 
easier.

> Will someone please write and submit a derivative that stores the data
> using XDR?

Ill do that immediately, once the code compiles with g++ v3.x . It 
should not take longer than an hour then. I don't want to submit code 
though that I cannot test because the library does not compile.

>>>> 2.) The second problem is speed when serializing large containers of
>>>> basic data types, e.g. a vector<double> or ublas vectors and 
>>>> matrices.
>>>> In my applications these can easily by hundreds of megabyte in size.
>>>> In
>>>> the current implementation, serializing a
>>>> std::vector<double>(10000000)
>>>> requires ten million virtual function calls. In order to prevent 
>>>> this,
>>>> I propose to add extra virtual functions (like the operator<< 
>>>> above),
>>>> which serialize C-arrays of basic data tyes, i.e. functions like
>>
>>> Serialization version 6 which was submitted for review includes
>>> serialization of C-arrays. It is documented in the reference
>>> under the title "Serialization Implementations included in the 
>>> Library"
>>> and a test case was added to test.cpp.
>
>> Yes, but it does so by calling the virtual operator << for each
>> element, which is very slow if you
>> call it millions of times.
>>
>>>> In conjunction with this, the serialization for std::vector and for
>>>> ublas vectors, etc. has to be adapted to make use of these optimized
>>>> serialization functions for basic data types.
>>>
>>> The library permits override of the included implementations.
>>> Of course,  this has to be up to the person who finds the
>>> the included implementation inconvenient in some way as he is
>>> the only one who knows what he wants changed.
>
>> That will not work since overriding is a compile-time decision while I
>> decide the archive format at runtime and thus need to have these
>> optimized functions available as virtual functions.
>
>> have you considered trying to provide your own specialization for the
>> following operator ( and << as well)
>
> template<class T>
> inline boost::basic_iarchive & operator>>(boost::basic_iarchive &ar, T 
> &t){

I am confused by that answer. Could you please elaborate?

Thanks in advance,

Matthias