Re: [boost] RE: Serialization Library Review

From: Matthias Troyer (troyer_at_[hidden])
Date: 2002-11-24 07:50:25

On Friday, November 22, 2002, at 10:07 PM, Dave Harris wrote:

> In-Reply-To: <CEC3AC40-FB8D-11D6-B4E0-0003939BD83A_at_[hidden]>
> On Tue, 19 Nov 2002 08:09:13 +0100 Matthias Troyer
> (troyer_at_[hidden]) wrote:
>> The only solution which comes to my mind is additional virtual
>> functions for writing blocks of primitive types, which default to
>> just calling the operator<< (>>) n times, but can be overridden
>> by optimized functions for those archive types where optimized
>> writes (load) are possible. Nothing else would need to be changed.
>> What is objectionable in that approach?
>
> Well, it is a lot of new virtual functions - one per type, almost 
> doubling
> the size of the archive interface. Is it worth it?

If we want this serialization library to be useful also for 
data-intensive computing we need to make sure that there we do not 
incur huge performance penalties. One call to memcpy, copying megabytes 
of data will surely be much faster than millions of virtual function 
calls, each copying just a few bytes.

>
> I am inclined to say it is, because I generally favour giving the 
> archive
> more information about what is going on. And efficiency matters,
> especially for such a fundamental library as serialisation. It does not
> lead to more work for serialisation users or for the archive 
> implementers.
> Still, it is a lot of new functions, and once added they can never be
> taken away. It may be better to leave them out now, and add them in 12
> months time, when we have had more experience with the library and its
> cost.

While I can accept your arguments I do not believe this will really 
help. Unless serialization of large collections of primitive types are 
efficient, nobody with that need will use the library and you thus 
cannot gain experience.

My rationale for including them right away is that for many formats 
there are functions which implementors of an archive might use and 
which have optimized versions for large arrays. Examples are,
XDR for portable binary archives
PVM and MPI for message passing
fwrite and fread for writing native binary files
memcpy for writing into memory-based buffers
Essentially all archive formats except for text-based ones can profit 
from these optimized functions.

Matthias