[boost] Serialization Library Review

From: Robert Ramey (ramey_at_[hidden])
Date: 2002-11-17 13:08:31

Date: Sun, 17 Nov 2002 10:19:23 +0100
From: Matthias Troyer <troyer_at_[hidden]>
>> From: Matthias Troyer <troyer_at_[hidden]>
>
>> Suppose you have a number on the first platform that exceeds 32
>> significant bits.  What happens when the number is loaded onto
>> the second platform.  Are the high order bits truncated? How
>> do you address this problem now?  If none of your longs
>> are larger than 32 significant bits then there is not problem.
>> If some are, the 32 machine can't represent them.

>> This can't cause any problems you don't have already.

>How do you do that when the bit size can vary from platform to  
>platform? Do you check at runtime what the number of bits is and  
>dispatch to serialization for that number of bits?

portable archives, e.g. text convert binary data to some canonical
form that ( e.g. arbitrary length strings) on save and back to 
native format on load.  So the only problem can occur if the
source machine can represent a number that the destination
machine cannot.  However you are handling this problem now
is how you will have to handle it in the future.  This cannot be
addressed by the serialization system.

>No, it seems that in the binary file you just write out the sizes of  
>the integers and just fail the loading if the bit numbers don't agree.  
>Using the fixed-bit-size integers instead would allow your binary files  
>to be much more portable.

It is mentioned in several places in the code, docs and in this list
that the native binary archive derivations have absolutly no 
pretentensions to portabability.  ABSOLUTELY NONE.  Knowing that
many users will ignore this admonishment, I included code
do double check that at least this size and endian ness of
the data types match between the loading and saving machine
so that the most obvious blunders will be detected and an exception be
thrown. I don't know what else I can do. The current
library has NO instance of a portable binary archive.  Many have
written to tell me how easy it is but I'm still waiting for someone
to submit one.

>As I mentioned in the introductory part of my post, text archives are  
>much longer than binary ones and thus cause bandwidth problems for some  
>applications. Note that the option to compress the archive after  
>writing works a) only if you serialize into files (which is only one  
>use) and b) does not address the bandwidth problem of first writing the  
>large text files.

Will someone please write and submit a derivative that stores the data
using XDR?

>> 2.) The second problem is speed when serializing large containers of
>> basic data types, e.g. a vector<double> or ublas vectors and matrices.
>> In my applications these can easily by hundreds of megabyte in size.  
>> In
>> the current implementation, serializing a  
>> std::vector<double>(10000000)
>> requires ten million virtual function calls. In order to prevent this,
>> I propose to add extra virtual functions (like the operator<< above),
>> which serialize C-arrays of basic data tyes, i.e. functions like
>
>> Serialization version 6 which was submitted for review includes
>> serialization of C-arrays. It is documented in the reference
>> under the title "Serialization Implementations included in the Library"
>> and a test case was added to test.cpp.

>Yes, but it does so by calling the virtual operator << for each  
>element, which is very slow if you
>call it millions of times.

>> In conjunction with this, the serialization for std::vector and for
>> ublas vectors, etc. has to be adapted to make use of these optimized
>> serialization functions for basic data types.
>
>> The library permits override of the included implementations.
>> Of course,  this has to be up to the person who finds the
>> the included implementation inconvenient in some way as he is
>> the only one who knows what he wants changed.

>That will not work since overriding is a compile-time decision while I  
>decide the archive format at runtime and thus need to have these  
>optimized functions available as virtual functions.

>have you considered trying to provide your own specialization for the
>following operator ( and << as well)

template<class T>
inline boost::basic_iarchive & operator>>(boost::basic_iarchive &ar, T &t){

Robert Ramey