Re: [boost] What should Serialization do?

From: Dave Harris (brangdon_at_[hidden])
Date: 2002-12-14 08:48:37

In-Reply-To: <20021214010158.95727.qmail_at_[hidden]>
On Fri, 13 Dec 2002 17:01:58 -0800 (PST) Augustus Saunders 
(infinite_8_monkey_at_[hidden]) wrote:
> I think that the points below can have multiple solutions, and that
> the points are orthogonal.  Different formats should be able to mix
> and match desired solutions.

Yes, and transparently to the user-defined types. I hope that UDTs will be 
able to see a single, common interface. Part of the trick of designing 
this is to avoid giving too many choices to the UDT.

> Presuming the deserializing application is written in a language that
> supports aliases.  A minor point, but this kind of point caused our
> serialization vs persistance discussion--serialization is allowed to
> be lossy and not support some language features.  Persistance would
> guarantee this in all cases, whereas it would depend on the format
> when serializing.

I meant things which should be supported when desired, not things which 
should be required by all formats.

The "serialisation vs persistence" terminology just clouds the issue in my 
view - it's enough that we agree that some formats should be allowed to be 
lossy.

> I want to provide a service for gathering meta-data, but let formats
> store it however they please.

Of course. The point is the API needs to distinguish between per-class 
data and per-instance data.

I am not sure if we need to support arbitrary user-defined meta-data. Once 
we have version numbers, object begin/end, and field identifiers, is there 
anything else we need? The things I mentioned have semantics which the 
framework needs to understand, so deserve special support. It is tempting 
to generalise beyond them, but there's a danger of over-complexification. 
Often the version number is enough.

> I can't tell if you want third parties to create formats, or if you
> want a serialization library to provide them.

Both. That's the approach of the current submission, and it seems 
obviously the right thing to do, so I've been taking it for granted.

> It's my intent to let other people write formats (except for a
> few examples).

Of course other people must be able to write new formats. We can't hope to 
cover all possible formats. However, I think boost ought to provide 
several formats out of the box. Not so much as examples, but as reusable 
best practice solutions, to save users the effort of doing it themselves. 
And in some cases to help promote cross-application and/or cross-platform 
cooperation.

The formats could include:

o  A reasonably portable binary format, which uses variable length
   integers to solve byte order and integer size issues. Metadata
   is pretty much limited to per-class version numbers.

o  A non-portable binary format, if the above is too inefficient.

o  A cut-down XML-compatible format. This would include meaningful
   tags but I wouldn't bother doing a full validating XML parser.

o  A "safe" text format, for channels which cannot pass binary,
   and where the verbosity of XML is not needed.

o  A debugging dump format. This would be more human-readable than
   XML, but there would be no way to load it.

I suppose this could be reduced to 2, portable binary and XML, at a pinch. 
However, I expect the others will get written and they might as well be 
put somewhere public and reusable like boost.

We can have some variations on the main themes. For example, it'd be 
pretty easy to give the portable binary format the option to store a 
single version ID for the archive rather than one per class.

-- Dave Harris