Re: [boost] [serialization] fast array serialization (10x speedup)

From: David Abrahams (dave_at_[hidden])
Date: 2005-11-17 18:41:19

"Robert Ramey" <ramey_at_[hidden]> writes:

> Matthias Troyer wrote:
>
> I'm sorry if that's how it came off.  It certainly wasn't directed at you.
> I crafted the email in response to Dave's request to take another
> look at it.  I resented being pressured into that was my response.

  ...

> Now I know you don't
> think its obvious or even correct, but we're not going
> to convince each other so there's no practical alternative
> to just letting things simmer a while until someone with
> a fresh perspective can make a case that can convince
> one of us to change is viewpoint.   Veiled threats to
> fork the library and make my life even more difficult are
> really way out of line and that's what I was responding to.

Robert,

This should have come sooner, but it's been part of a longer
communication that I've been working on all week and, I'm afraid,
isn't going to get finished all in one message.  So I'm sending the
first part now because it should have gone out long ago.

First of all, please accept my apology.  I didn't mean it as a threat,
but on re-reading it's obvious to me why it sounded that way.  In fact
I can't see how anyone could have interpreted it differently without a
lot more context.  I hope by the time you finish this series of
messages that will be possible, but regardless, again please accept my
apologies.  The pressure was, as you say, out of line.

Also, I'd like to acknowledge that without your tenacity, dedication,
expertise, and, above all, your desire to meet the needs of the user
community, this library never would have made it through _two_ formal
reviews and been accepted into Boost.  From reading feedback on the
mailing list it has quickly become one of the most-used libraries in
Boost, so it is obviously a major contribution to the C++ community at
large.  Your continued stewardship is crucial, and appreciated.

I've been traveling this week and thinking carefully about Matthias'
design and your concerns with it, and -- believe it or not -- I think
we can offer a fresh perspective.  It took me some time to understand
your concerns about intrusiveness and coupling, but I think I do have
a handle on it.  In my opinion, Matthias' design was a bit more
complicated than necessary, which can't have helped you to feel
receptive.  After some work, we think we have a basis for discussion
that _begins_ with a design that meets all your criteria for approval.

Introduction
============

In an upcoming message I'm planning to start by describing the least
intrusive design that could possibly work -- one that makes no changes
at all to the existing serialization library.  It's not a bad design,
but it has a few drawbacks that I'd like to discuss.  At that point it
should become clear what I meant about "hijacking" the serialization
library.  Finally, I'll describe the smallest set of changes to the
existing serialization library that would be needed to address those
drawbacks.

Just to state the obvious, I hope to convince you that those few
changes are worth making.  Of course, as the maintainer of
Boost.Serialization, it's completely up to you whether to do so.

Before getting into design details, let me highlight the reason for
this proposal in a way you may not have seen it stated before:

,----
| For many archive formats and common datatypes there exist APIs that
| can quickly read or write contiguous sequences of those types all at
| once (**).  Reading or writing such a sequence by separately reading
| or writing each element (as the serialization library currently
| does) can be an order of magnitude more expensive.
`----

We want to be able to capitalize on the existence of those APIs, and
to do that we need a "hook" that will be used whenever a contiguous
sequence is going to be (de)serialized.  No such hook exists in
Boost.Serialization.

(**) Note that this capability is not necessarily tied to bitwise
     serialization or the use of a binary representation.

The Design
==========

We've attempted to use programming idioms and terminology found
in the existing serialization library wherever possible, so that it's
easy for you to read and understand, and you won't be distracted by
minor stylistic differences.

In the messages to follow, the word "array" will normally mean a
contiguous sequence of instances of a single datatype, and not to a
C++ builtin array type of the form T[N].  I'll try to be explicit when
I intend to describe builtin arrays.

<more to come>

-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com