Re: [boost] [serialization] fast array serialization (10x speedup)

From: Manfred Doudar (manfred.doudar_at_[hidden])
Date: 2005-11-24 21:30:03

Robert Ramey wrote:
> Ian McCulloch wrote:
> 
> 
>>Besides, is the boost iostreams library really much slower than a
>>hand-coded buffer?
> 
> 
> I'm pretty confident that its much, much slower, but this will remain
> in dispute until someone runs the code with a profiler.
> 
> 
>>Anyway, this is a side issue.  The main point is:
>>
>>David Abrahams wrote:
>>
>>>  ,----
>>>  | For many archive formats and common datatypes there exist APIs
>>>  | that can quickly read or write contiguous sequences of those
>>>  types | all at once (**).  Reading or writing such a sequence by
>>>  | separately reading or writing each element (as the serialization
>>>  | library currently does) can be an order of magnitude more
>>>  | expensive.
>>>  `----
> 
> 
> Sorry - that's NOT the main point.
> 
> The main point is - do enhancement for special cases have to be
> incorporated into the the core code so that everybody else
> is obligated to use it?  What are the advantages and
> disadvantages of doing so?

Umm, I don't follow - how are others going to be obligated to use those 
special cases that don't necessarily apply to them?

> 
> 
> No one is disputing that it desireable to be able to extend
> the library for these special circumstances.
> 
> 

Actually it's *very* desirable.

> 
>>If there is to be any possibility of targetting an archive to this
>>format, then array support is crucial.
> 
> 
> Then just make an archive which does it- what's stopping you?
> 

True - but the suggestion is that it's common enough to want as part
of the core library.

Here we use netCDF (the format brought up by Ian) and HDF-5 data files, 
and run with MPI on a mosix cluster ...

the point being that these data types are integral to industries like 
ours - where large volumes of data are the norm - and array access is 
indeed crucial - we see code here that runs in the order of seconds to 
minutes, or hours depending on how data is read and written - but as a 
concesion, that is also dependent on the underlying libraries we inherit 
from 3rd parties (aka netCDF, HDF-5, et al).

I've been loosly following this thread and I understand your reluctance 
to this - but it's a matter of perception as to how needful, and 
widespread such a need is.

... back to lurker mode

Cheers,
-- 
Manfred Doudar
MetOcean Engineers
www.metoceanengineers.com