Re: [boost] [serialization] Proposal for an extension API to the Archive concept

From: Robert Ramey (ramey_at_[hidden])
Date: 2007-09-28 12:04:48

> I've tried to describe the options as dispassionately as I could, so
> as to
> lay a common ground for further discussion. Do you see any error
> in the description? Are you satisfied with this rendering of the
> available
> options?

A very good summary and explanation.

> But an important one (and more are coming, e.g. std::tr1::shared_ptr).
> You are of course free to avoid expanding the Archive concept as
> in E) and F), but the logical implication of this is that shared_ptr
> is
> technically not serializable. Nothing wrong about that, if that's your
> declared intention, but users should know.

OK - now I see better your concern.  I would say:

shared_ptr as concieved and implemented by boost does not provide
sufficient exposure to be support the concept of Serializable Type
as defined by Boost Serialization:

Given its importance, various hacks have been made to address it,
but they are still basically hacks.  They've resulted in code which
requires coupling of the two classes/concepts.  The current
and proposed resolution of this situtation requires is at best
inefficient.  The original implemenation of serialization for
boost smart_ptr, was more twice as fast but depended on the
implementation.  The current one is slower, but but depends
only upon the published interface.

To really resolve this situation in a definitive manner, either
Serializable/Archive Concept or shared pointer public interface
have to modified.  In effect our hack for shared_ptr extends
the Archive Concept for this one specific case.  But at least
its optional and extends the concept by composition so that
the original Concept remains intact and the hack is only
visible, confusing, illlogical and opaque in the context of that
one type.  And someone comes up with a serializable shared_ptr
then this can be dropped without having left any collateral
damage in the Archive Concept.

shared_ptr is the only type which has come up in several years
which has this problem.

Some application specific types might have this issue, but in those
cases, I would expect the ability to attach an application specific
helper to the archive, thereby expanding the Archive Concept
for just that application would be acceptable.

Of course these observations would apply to TR1::shared pointer
as well as other serialization systems.

>> Another layer of concept for users to (mis)understand and ask about
>> and whatever.
>
> That's the job of a lib maintainer.

LOL - well, that's me. And really I'm not only trying to keep the job
from getting bigger, I'm trying to make it smaller.

> I hope you assess every possible
> extension to your library on the basis of its technical merits. If
> you plainly state
> that B.S is closed for extension, well, this discussion is moot.

The power of any software (or mathematical/logical ) module resides
in its logical coherence.  Arbitraritly extending it in tangent directions
requires that the a use of the module or concept consider a bunch
of special cases every time he uses even for simple cases.  Such
types of extensions result in a net reduction in the utility of the library.

The best way to extend this library (or any other) is by adding
specific behavior/facilities on the current one.  C++ supports
this through inheritance.  My latest attempt at handling the
shared pointer issue reflects this point of view.

Remember, this whole issue has come about because shared_ptr
(unlike any other type so far) has been written in a manner that
it is effectively closed for extension.  Maybe you want to direct
some observations in that direction.

I've just come accross a very relevent instance of this.  the 1.35
version of the binary

In order to support optimization of a couple of collection types
extra machinery was added to basic_binary_?archive (in 1.35)
This was subject of a fairly acrimonious discussion last year.
This was packaged along with some needed improvements
in the handling of collection.  This enhancement basically
breaks the Archive Concept for these types.  I actually
overlooked this aspect at the time. But in the last month
in the course of doing an exhaustive test on portable_binary_?archive
I found that it was broken.  This was due to the fact
that it was derived from binary_archive and
that binary_archive interface was extended at the base
level to handle these special cases.  Among other things,
this made portable_binary_archive useless as an
example of how to extend and archive through derivation.
So now I have to drop the example from the documentation
and conjure up a new one, explain why extension
through deriviation shouldn't be done for binary archive
and make a new example based on another archive class.
A huge net loss.  And I had to make a new portable
binary archive which didn't derive from basic_binary_archive.
I had to spend a whole day to discover the problem
which only showed up on a couple of tests.   All this
from a simple optimization which "everyone" needs.

>> And there will always be data types whose authors
>> don't wish to expose enough functionality to be compatiable with
>> boost serialization - so were does it end?  And all this for just
>> one such datatype so far?

> In my particular case, I can assure you that the need of a helper API
> is a genuine one: I cannot serialize my type efficiently without that
> API,   no matter how much I want to expose the guts of my type for
> serialization  purposes. Please believe me on that, this is not a case of 
> me wanting
> to  keep my type encapulated, "pure", or anything. It's sheer
> impossibility  to do it otherwise.

Hmm - on one hand, I don't doubt your sincerity nor your assessment.
On the other hand, I've yet to see such a type myself.  The original
serialization of shared_ptr was implemented by exposing more of
the shared_ptr internals so in that one case it is possible.  Of course
whether that solution is desirable would be a matter for discussion.

> Of course, coming up with some more types that could put the helper
> API to good use would strengthen my position, that' for sure.

So the current situation, is

a) We have an Archive Concept, and Serializable Concept which are
fairly coherent.  The are currently only broken by basic_binary?archive.

b) The classes in the library common_?archve, basic_?archive, are
implementation features.  It is not required to derive from any of these
classes to implement the concepts.  Of course its convenient to do
so as they implement common aspects of the Archive Concept - but
they are not required to.

c) Some types - so far only shared_ptr and your ? - do not conform
to the concept of a serializable type as they stand.  My position is

i) they are very infrequent.
ii) the Archive Concept can be extended for these special cases through
composition (inheritance) to provide ad hoc solutions.

That's what 1.34 did for shared_ptr and I'm willing to continue doing that.
The only think i want to do is to move the helper API out of the 
basic_?archive
where it pollutes the Archive concept (which is why i didn't document it)
and package it as a mix-in which is used with naked_?archive to produce
the "shared_ptr" friendly archive classes.

What do you see wrong with that?

Robert Ramey