Re: [boost] array optimized serialization

From: Robert Ramey (ramey_at_[hidden])
Date: 2006-09-21 18:18:53

Matthias Troyer wrote:
> On 21.09.2006, at 16:34, David Abrahams wrote:

> Robert, you did not have any complaints on June 4th, but just asked
> me to add our optimizations directly to your binary archive.

Its entirely possible I might have suggested this without correctly
envisioning what the final result would look like.
uhhh - that happens to me all the time.  I suppose that's why
I'm a software developer as opposed to say ... - a diamond cutter
or suicide prevention counseler.

> If you
> feel more comfortable without this, we can undo this at any time
> since it will not break any existing archives. If however, as you
> write, you feel you can live with what we have done then we can leave
> it as it is and I can certainly take over responsibility for
> maintaining the array optimizations and any further extensions to it
> that might be desired.

That's fine with me - really.

My concern is that the maintainence and support of this has been
underestimated.   Personally I've found the code very hard to follow.
However, It's  not a big issue for me now that you've offered
to do it.

At the risk of picking at a festering sore - here is my (imperfect)
recollection of how we got here.

The original proposal required adding save_array to all present and
future archives.  I objected to the idea of expanding the interface
just to accomodate a special case.  This was resolved with the
"array_wrapper" which provided an overridable default which
was suitable for "other" archives.  This was a HUGE improvement
as far as I was concerned.

As things progressed, it became clear that support for"serialization 
wrappers"
- including the new array_wrapper - needed to be generalized
some in order to remove direct dependency of the serialization code
on certain specific types (such as nvp and array, and C++ arrays).
This you did - Another significant improvement.

So now we have a specific wrapper (array_wrapper) handled in a
specific archive binary_archive.  This is analogous to the situation
in xml_archive where nvp is handled in xml_archive.  (Its not quite
the same - as xml_archive without nvp really makes no sense.)
To me the current implementation stops short of carrying the
wrapper idea to its logical conclusion. - having the binary archive
code independent of array_wrapper (or any other particular wrapper
for that matter).  This is in line with my (mostly successful)
campaign to keep archive code independent of particular data types.

I also believe that things have been confused by an evolution
in the original idea for save/load array.  As I remember the motivation
was that save/load array could be re-implemented differently for
binary and/or mpi and/or other archives.  Of course we have this
now as the array_wrapper can be implemented differently for
each archive.  But now it seems that save/load array are going
to be quite different for binary and mpi type archives.  I think that
at one point there might have been the idea that save/load implementations
could be shared between different binary-type archives.   But now
it seems that binary_archive an mpi and/or other archives will have
less in common than originally thought.

I think this evolution has led to code which is very hard for me to follow
and which will be more work to maintain and support. I can't help
but think that if 10x speed up of binary_archive were considered today,
(given the wrapper and other changes), the enhancement would
be less general and much simpler.

Good Luck

Robert Ramey