Re: [boost] [serialization] fast array serialization (10x speedup)

From: Peter Dimov (pdimov_at_[hidden])
Date: 2005-11-24 13:27:35

David Abrahams wrote:
> "Peter Dimov" <pdimov_at_[hidden]> writes:
>
>>> What if the number n is infinite (e.g. all possible structs
>>> consisting only of fundamental types), which is what Robert calls
>>> "bitwise serializable"?
>>
>> Structs aren't bitwise serializable in general because of
>> padding/packing/alignment. Archives that do not have a documented
>> external format and just fwrite whatever happens to be in memory at
>> the time aren't really archives, they are a very specific subset
>> with limited uses (interprocess communication on the same machine,
>> the same compiler and the same version) that should not shape the
>> design.
>
> If Stepanov had used that philosophy we wouldn't have algorithms
> specialized for random access iterators.  Containers with random
> access are "a very specific subset" with, arguably, "limited uses"
> (don't forget that the original Lisp guys thought it would be better
> if everything were made up of cons cells).  The ability to specialize
> generic algorithms to take advantage of special properties of
> "specific" datatypes is fundamental to Generic Programming.
>
> In fact, every example we can think of so far where optimized array
> serialization is useful is just such a "specific" archive.

An archive turns a C++ data structure into an untyped stream of bytes in a 
reversible way. The specific way to map C++ into bytes (the external format) 
is what distinguishes one archive from another.

There is _one_ archive for which the external format is the same as the 
memory layout. It's possible to play with the serialization of 
non-contiguous data structures and create several such archives for the sake 
of NIH, but all these archives are isomorphic, they conceptually represent a 
single point in the design space.

Whereas there are a number of distinct (non-isomorphic) random access 
containers.

>> ("Archive" implies persistency, and relying on a specific
>> memory layout is not a way to achieve it.)
>
> I think Robert's statement
>
>  "Here, we use the term "serialization" to mean the reversible
>  deconstruction of an arbitrary set of C++ data structures to a
>  sequence of bytes. [...]

Robert's statement is not at odds with what I wrote above.

>> If you have such an archive, you can add an overload SFINAE'd on
>> is_bitwise_serializable
>
> Robert wants portability to vc6, which doesn't support SFINAE.  I
> doubt he'd want to accept a change that, to be practically taken
> advantage of, would require users to apply SFINAE.

On VC6 you can use separate overloads.

I have the feeling I must be missing something fundamental. :-) What do you 
perceive as the important difference between providing an optimize_array 
overload that returns mpl::true_ and providing a save_sequence overload that 
calls .save_array? (Except that the latter is obviously more flexible.)