Re: [Boost-users] [serialization] skipping data

From: Joel de Guzman (joel_at_[hidden])
Date: 2006-07-10 19:02:18

Peter Dimov wrote:

>>> The option of actually reading a temporary Y and destroying it
>>> afterwards is also feasible. You don't necessarily have to optimize
>>> the new version so that it reads v1 files faster than v1 itself. :-)
>> There might be a possibility that Y is already obsolete at
>> version 2.
> 
> By basing an external file format on Y's internal structure, you've ensured 
> that Y never becomes obsolete. :-)

Unless, you allow for skippable archives, like IFF does.
In effect, what you do is provide a nullable abstraction
to your data structure. Skippable archives allow data to
be obsolete.

>> Sure you can "emulate" it through the proxy thing,
>> but then again that's less than ideal. I simply want to skip
>> some data and I can't do it with Boost.Serialization. I can
>> do it with simple streams, for example, by prepending the data
>> with a length (in bytes). I can also do it with XML by ignoring
>> everything in between the current tag and its matching end tag.
>> Rene (Rivera) noted that some archive types also allow it. For
>> example, IFF (http://en.wikipedia.org/wiki/Interchange_File_Format)
>> allows it through chunking. "Because the spec includes explicit
>> lengths for each chunk, it is possible for a parser to skip over
>> chunks which it either can't or doesn't care to process."
> 
> You are right that when the file format has a "shell" structure that 
> encompasses the serialization format, it is possible to skip over parts of 
> the stream. To write IFF, however, you either need a seekable stream, or you 
> need to serialize the chunk contents into a temporary archive in order to 
> obtain its size. I'm not sure whether the current serialization framework 
> can support something like that.

Yeah, that's the problem.

> I can't help but wonder: if some parts of your v1 format are so completely 
> redundant as to allow you to reconstruct the original data even if you skip 
> them, what was the point of writing them in the first place?

Here's one scenario (I'm sure there are others): Your class uses a
3rd party library called Y. Later, you decide to replace it with
a better engine using a library called Z.

Regards,
-- 
Joel de Guzman
http://www.boost-consulting.com
http://spirit.sf.net