Re: [Boost-users] [Serialization] Writing an old version of a class

Subject: Re: [Boost-users] [Serialization] Writing an old version of a class
From: Kerry, Richard (richard.kerry_at_[hidden])
Date: 2011-07-26 12:15:34

This discussion of handling of old vs new versions of data structures reminded me of a system we wrote at a place I used to work.
The system which was able to be both backward and forward compatible, within reason.

What we did was to write the data to the file with a numeric identifier, or tag, and a data length.  

The rules were as follows:
Once you've defined the contents of a particular tag you were not then permitted to add or change anything within the data structure that it wrote, but you could add to the end.
The reader must read all tags.  It may skip any it doesn't recognize (ie it mustn't consider that to be an error). 
It will use its own data structure according to the current version of software.  It will assign default values to all parts of all structures before reading anything from the file.
It will then read a maximum number of bytes from each record in the file according to the length of its own version of the structure.
If there are not enough bytes then the trailing part will retain the default values.  If there are too many they will be discarded.

This achieves conventional backward compatibility (ie new software can read data from old file) because the new software will :
Have default data for (the trailing part of) structures which were too short in the file.  Ie written by older versions.
Have default data for structures which were not present in the file (ie the tags weren't found).  Ie not written at all by older versions.

It achieves forward compatibility (ie old software can read data from new files) because the software will :
Ignore data in the record that is past the length the (old) software expects.
Ignore any tags it doesn't recognize.

I do admit I'm writing this without much knowledge of Boost's serialization.
Reading this thread reminded me of this system - maybe Boost can already handle this...
And maybe it's an idea to be considered if not.

Regards,
Richard.

 
 

> -----Original Message-----
> From: boost-users-bounces_at_[hidden] 
> [mailto:boost-users-bounces_at_[hidden]] On Behalf Of Don
> Sent: 16 July 2011 19:40
> To: boost-users_at_[hidden]
> Subject: Re: [Boost-users] [Serialization] Writing an old 
> version of a class
> 
> -------- Original Message --------
> Subject: Re: [Boost-users] [Serialization] Writing an old 
> version of a class
> From: Robert Ramey <ramey_at_[hidden]>
> To: boost-users_at_[hidden]
> Date: 7/15/2011 7:34 PM
> 
> > Noah Roberts wrote:
> >> On 7/15/2011 5:27 AM, Don wrote:
> >>> I apologize if this is a duplicate posting. I sent this 2 
> days ago,
> >>> and it appears in the archives, but I never received it back from
> >>> the list and there have been no replies, so I am posting it again.
> >>>
> >>> Using boost serialization, I would like to be able to write an old
> >>> version of a class, so that I can save data in an "old" format to
> >>> allow other users with an old version of my program to 
> read the file.
> >>>
> >>> I see from the archives that this has come up before, and Robert
> >>> Ramey indicated it was doable and on the todo list, but I 
> don't see
> >>> it in the release notes or documentation. Did anything 
> every come of
> >>> this?
> >>
> >> As far as I had been informed, backward compatibility in 
> writing was
> >> never planned for the serialization library and was never 
> even planned
> >> to be planned.
> >
> > That's exactly true.  But I did look into it and much to my surprise
> > I did conclude that there was no fundamental obstacle - just quite
> > a bit of work.  Enhancing the the serialization library - 
> or even fixing
> > a bug - is more problematic than with other libraries 
> besides avoiding
> > breaking interfaces, one needs to be able to read previously created
> > archives as well.  Then there is an issue that implementing 
> this exposes
> > a few asymetries in the implementation which would also have to
> > be addressed somehow.  All in all quite a job.
> >
> >> I don't think you'll find a way to do it without
> >> severely altering the library itself.
> >
> > It wouldn't be THAT sever - BUT, even a small change has quite
> > a bit of "ripple effect"
> >
> >> Personally, I think this is a significantly lacking feature and I'd
> >> love to see it implemented.  If you pull it off, please consider
> >> sharing.
> >
> > It never occurred to me that such a thing would ever be useful.
> > Also it isn't requested very often.  I'm not saying that it's
> > an illegitimate request - just one that no one has found it
> > so necessary as to actually invest the required effort.
> 
> Thanks for the update, Robert. It's a requirement for me, so 
> I guess I'm going to have to figure this out or find an 
> alternative solution. (As far as I'm concerned, this is the 
> biggest failing of MFC serialization, too.) If anyone knows 
> of other tools for this, please let me know.
> 
> FWIW, I believe the change is a little more complicated than 
> you indicated in your 2007 postings on this subject to this 
> list, or perhaps I don't really understand your comments 
> there. It seems to me that you need to keep a table of which 
> object versions correspond to the file you want to write. 
> Consider the situation where you are working on version 2.0.0 
> of your application MyApp. You want to be able to save MyApp 
> files both in version 2.0.0 format, and also in version 1.0.0 
> format, so you can send files to other users who still have 
> version 1.0.0 of MyApp. (Imagine you are coding Word 2007 and 
> you need to save files in Word 97 format.) When saving in 
> version 1.0.0 format, you need to know what version of each 
> object you need to save. You will have to create that table 
> of object version numbers (presumably by hand), but how does 
> the serialization system access that table in order to be 
> able to pass the correct version number to each object's Save 
> function? Or, the serializ
> ation system needs to pass a *file* version number (not 
> object version number) to the Save functions, and they need 
> to translate that to the appropriate object version number 
> themselves. This might be easier, but it loses the symmetry 
> between load and save, and it distributes the file-version to 
> object-version relationship across all the objects.
> 
> Thanks,
> 
> Don
> 
> 
> 
> 
> 
> _______________________________________________
> Boost-users mailing list
> Boost-users_at_[hidden]
> http://lists.boost.org/mailman/listinfo.cgi/boost-users
>