Re: [boost] [repost][serialization] generic runtime helpers

Subject: Re: [boost] [repost][serialization] generic runtime helpers
From: Robert Ramey (ramey_at_[hidden])
Date: 2008-09-04 12:52:11

joaquin_at_[hidden] wrote:
> Hello, allow me to repost the following lest my first try went
> unnnoticed:

I didn't go unnoticed, its just that its not easy to answer.

> The section on proposed case studies for Boost.Serialization talks
> about the poosible inclusion of a so-called "generic runtime helper":
>
> http://tinyurl.com/6khxns
> http://www.boost.org/doc/libs/1_36_0/libs/serialization/doc/new_case_studies.html#archivehelper
>
> which subject I had the pleasure to discuss with Mr. Ramey some
> time ago. I've got a couple of questions regarding this feature:

> 1. Is there any tentative release version for getting this included?

no

> 2. Is the idea to include the generic runtime helper as part of the
> core API of an Archive, or will it be just an adaptor? In other
> words, would the core "Saving Archive" and "Loading Archive"
> concpets be extended to also feature this runtime helper?

I've considered that and don't think its a great idea.  I looked over
the code in flyweight and concluded that, even though the usage of
a helper might make the loading of an archive slightly more efficient
it would not be strictly necessary.

> FWIW, I'm strongly in favor of having the runtime helper be part of
> the core Archive concepts: it is my stance that this can be incredibly
> useful to implement serialization code for types with special
> lifetime semantics (as is the case with boost::shared_ptr
> and boost::flyweight) without violating the encapsulation either of
> the  type or of Boost.Serialization. The key and only property of such a
> runtime helper is its ability to associate an Archive object with a
> type-dependent helper object, which I see as a very modest, yet
> highly useful, extension to the Archive concept.

> Just to exemplify how useful helpers can be, allow me to sketch the
> serialization process of flyweight<T> as implemented with runtime
> helpers:
>
> * Saving: the helper<flyweight<T> > is just a bag of flyweight<T>s
> whose elements are *exemplars*. when saving a new flyweight,
> first we check whether an equivalent flyweight is already present
> in the helper: if so, the index of the exemplar is output, otherwise,
> the underlying T associated to the flyweight is saved and the
> flyweight is added to the helper as a new exemplar.
>
> * Loading: again the helper is a bag of flyweight<T>s. When
> loading a flyweight<T>, we can read either an index or a T
> object: in the former case we just initialize the flyweight with the
> corresponding exampler; in the latter case we load the T object,
> construct a flyweight from it, add it as new exemplar and
> initialize the loading flyweight with it.
>
> This implementation is totally non-intrusive and maximally efficient.
> An observant reader will also notice that it can be translated
> almost literally to a serialization scheme for shared_ptr, which shows
> the potential of the runtime helper feature. Currently, shared_ptr
> serialization relies on a non-documented adhoc helper, which
> technically makes it a non-serializable type: promoting the runtime
> helper to the Archive concept would allow to make shared_ptr a
> first-class citizen. I also think more types can benefit from this
> feature when implementing their serialization code.

I don't dispute the utility in such cases.

Adding such a facility to the library has significant costs

> Valid concerns may be raised that extending the Archive concept
> would break backwards compatibility. However, it is easy to help
> programmers migrate their user-defined archives to the new
> concept: just provide a mix-in base class implementing the required
> functionality, so that if

....

I don't think this is a problem

Here is my objection.

a) So far with only one exception, all components of the standard library
and boost have been serializable types.  The only exception so far
is boost::(and tr1::)shared_ptr.

b) adding a generic helper to the library increases the "conceptual
surface area" by a significant amount.  That is, it adds a layer of
functionality which users may feel compelled to understand in order
to properly make use of the library.  All this effort for just one
and now one other non-serializable type. (which can be serialized
without this.

c) It adds a whole new layer of developement, testing, and documentation
which I don't want to undertake - again all for one new type.  Oh and
let's not forget the effort consumed when someone else want's to have
this or that different aspect.

d) Having such a facility along with the concept of "having things
in only one place" would create an imperative to re-implement the
current shared_ptr helper in terms of the new generic helper.  More
work.

e) Including such a generic helper would add a tiny amount of additional
overhead to every program on earth which uses serialization.  This
overhead would be in the form of additional code to manage a list
of attached generic helpers.  This extra code would be invoked only
by those programs which use such a helper.  But it would take up
space in every executable.  There have already been complaints
that the serialization library isn't light weight enough.

e) There is no obstacle to crafting a generic helper in such a way
that it can be applied to any archive by crafting and "archive adaptor"
template - along with examples, documentation, tests, etc.  There
is no reason that it has to be part of the "core" serializaton library.

So, I'm not interested in taking this on.

However, I can offer the following option.

a) craft a generic helper.  This shouldn't be tooooo dificult as such a
helper existed in 1.33.

b) craft an archive adaptor template.  This shouldn't be tooooo difficult
either.  The current helper used to make shared_ptr serializable should
be an inspiration.  This replaced the previously "built-in" functionality
with the current "added-in" functionality and cleaned up some
of the serialization library code.

c) Create documentation for the above.  Include as a case study "archive 
adaptor"
which describes archive adaptor for adding generic helper.  Include
some small tests to test the whole thing for all the archives.  You only 
have
to make one test for each concept as the serialization library test setup
will test it for each archive.

If, after a release or two, there is ground swell to include such a generic
helper in the core library concept, this could be discussed at the time. 
Were
this to be done at this point it would only mean altering a typedef and
tweaking the documentation to indicate that was previously an add
on is now added whether requested or not.

Robert Ramey