Re: [boost] boost serialization backwards compatibility of binary archives broken

Subject: Re: [boost] boost serialization backwards compatibility of binary archives broken
From: Robert Ramey (ramey_at_[hidden])
Date: 2013-02-20 17:00:58

Markus Henschel wrote:
> Hello,
>
> in its current state boost serialization isn't able to read binary
> archives from older versions. The documentation states that there is
> a problem with version 1.42-1.44 but this is not the whole story. My
> archives from 1.34 don't work either. There are currently open bugs
> for this:
>
> https://svn.boost.org/trac/boost/ticket/4903
> https://svn.boost.org/trac/boost/ticket/5567
> https://svn.boost.org/trac/boost/ticket/4660
>
> This situation has been like this for a very long time now. Comments
> of users even suggest not using binary archives or switching to
> another serialization library because of maintenance problems. I'm
> posting to this list as it seems like no one cares about these bug
> reports anymore. The least thing that should be done I think is to
> update the release notes. There are even patches in the bug reports
> that provide fixes although I don't know if they work for all
> situations. The current situation is quite uncomfortable for me as I
> have to patch every new boost version to get serialization working.
>
> What can be done to improve this situation?

I've left these open because I don't have a definitive fix.

This came about when I made some changes just to eliminate some
warnings when used for some compilers.  I considered these changes
inconcequential (just fixing warnings after all) so I didn't bother 
incrementing
the serialization library layout index in the header record.  Well, fixing
these warnings had some unintended side effects which changed the
archive format. This prevented reliable loading of binary archives for
previous version.  Of course this took a while to see where this was
coming from so even another version went by - this time with a number.
Serialization library testing was sufficiently exhaustive to detect this in
time and it took a while to pin down.  Naturally I expect to be much
more careful in the future.

I'm very much aware that this creates a problem for some users but
I've concluded that I can't make any change in the library that will really
fix it.

It's not totally hopeless however.  Would the following procedure work
for you?

a) using boost 1.34 make small program which
i)  loads the binary archives into memory  using boost 1.34
ii) saves the data loaded above  using text_archive or 
portable_binary_archive.

b) using boost 1.49+ make a small program which
i) loads the text_archive created in step a) above
ii) saves the data loaded above to a new binary_archive.

I believe this would permanently fix the problem.

And I promise to be more careful in the future.

A couple of misc notes on this subject.

a) this situation occured because I made a mistake while updated the
boost serialization library.  But it could occur from other sources - 
example
a compiler upgrade which changes the properties of some primitive datatype.
So I would recommend using text or portable binary archives for storage
to disk and reserver binary_archives for temporay archives.

b) I would like to enhance and make more robust the archive aspects
of the boost serialization library but I'm a little "gun-shy" after this 
experience.
The main problem is that I don't a good set of tests for backward
compatibility, making such tests would require a non-trivial effort,
and I feel the serialization library already places a disproportionately
large burden on the boost testing infrastucture.

This discourages me from adding some necessary enancements like
yaml, json archives and archives which create xml dtd/schema to permit
editing with xml tools and archives which save/load to/from GUIs
like mfc, qt, wxWindows, html5, etc.

I do maintain the library and have relatively few issues because it's
not hard to do.  Unfortunately, the best I can do for your issue is
what I've suggested above.

I hope you appreciate that it is embarassing for me to confess to all this.

Robert Ramey

> Thanks,
>
> Markus Henschel
>
> _______________________________________________
> Unsubscribe & other changes:
> http://lists.boost.org/mailman/listinfo.cgi/boost