Re: [Boost-users] [serialization] skipping data

From: Peter Dimov (pdimov_at_[hidden])
Date: 2006-07-10 11:58:39

Joel de Guzman wrote:
> Peter Dimov wrote:

>> This is the only way that works reliably with any archive. :-) (And,
>> since the serialization library does not document its external
>> format, the only way that works reliably with it.)
>>
>> The option of actually reading a temporary Y and destroying it
>> afterwards is also feasible. You don't necessarily have to optimize
>> the new version so that it reads v1 files faster than v1 itself. :-)
>
> There might be a possibility that Y is already obsolete at
> version 2.

By basing an external file format on Y's internal structure, you've ensured 
that Y never becomes obsolete. :-)

> Sure you can "emulate" it through the proxy thing,
> but then again that's less than ideal. I simply want to skip
> some data and I can't do it with Boost.Serialization. I can
> do it with simple streams, for example, by prepending the data
> with a length (in bytes). I can also do it with XML by ignoring
> everything in between the current tag and its matching end tag.
> Rene (Rivera) noted that some archive types also allow it. For
> example, IFF (http://en.wikipedia.org/wiki/Interchange_File_Format)
> allows it through chunking. "Because the spec includes explicit
> lengths for each chunk, it is possible for a parser to skip over
> chunks which it either can't or doesn't care to process."

You are right that when the file format has a "shell" structure that 
encompasses the serialization format, it is possible to skip over parts of 
the stream. To write IFF, however, you either need a seekable stream, or you 
need to serialize the chunk contents into a temporary archive in order to 
obtain its size. I'm not sure whether the current serialization framework 
can support something like that.

> IMO, this is a valid use case that Boost.Serialization can address,
> at least, for archive forms that allow it. Perhaps a SkippableArchive
> concept?

This could be possible if you don't mind constraining your class to only 
support SkippableArchives.

> IMO, the ability to skip is a prerequisite for transparent versioning.

Transparent versioning via skipping?

I can't help but wonder: if some parts of your v1 format are so completely 
redundant as to allow you to reconstruct the original data even if you skip 
them, what was the point of writing them in the first place?