Re: [boost] Serialization Formal Review #2

From: Matthias Troyer (troyer_at_[hidden])
Date: 2004-04-26 19:49:36

On Apr 24, 2004, at 8:18 PM, Robert Ramey wrote:

> Matthias Troyer wrote:
>
>> I have tried extending the library since I want to swap our old
>> serialization library used at http://alps.comp-phys.org with a Boost
>> serialization library as soon as possible. It was important to keep 
>> the
>> same archive format, since we have gigabytes of data collected over 
>> ten
>> years in millions of CPU-hours and need to be able to read those with
>> both new and old codes. With the help of Robert I managed to convince
>> myself that this is indeed possible
>
> Perhaps it might be possible, but I'm doubtful that it's the best way 
> to
> address the problem.  If the legacy format is a meta-data format it 
> can be
> possible in a way similar to the way the XML was handled.
> As file are processed they are automatically converted to the new 
> system.
> No special programming is required as the code to load the class 
> instances
> from the legacy data format already exists - it's the legacy code ! 
> its free
> at this point.

In our case this is not possible as long as applications using the old 
format are still in use. More than a dozen people at various places 
around the worlds have written simulation programs and produced codes 
written based on our application framework. I thus do not have full 
control over all the data that is around and need to support the legacy 
format for years to come.

On the other hand, I have full control over the future of the 
serialization level of these codes. The current version of your library 
allows me to implement an archive compatible with the legacy format. 
This will allow me to easily swap my serialization library with yours, 
and I need to support the legacy files just through a simple archive 
type. Old codes will still compile and might not even notice that 
serialization is now handled by your library. Application developers 
can then, if desired rewrite their codes to employ new features of your 
library, but they do not have to.

If I had all the data and all codes based on our libraries under my 
control, I could, as you recommend just change codes and file format. I 
wish to thank you for your advice and your willingness to help. What is 
most important for me (and maybe also useful for others) is that I can 
use your library and still read my old files.

>> 4. Documentation of archive formats, especially what class and object
>> information is stored. I want to be able to predict, from reading the
>> documentation, what exactly will be written to the archive and in 
>> which
>> order.
>> This is essential when exchanging data with an application that
>> does not use this library.
>
> I'm skeptical that this is going to be fruitful.  The "other" 
> application is
> going to have to effectively re-implement this whole library.  Why 
> bother,
> just use this one.

The other applications may be written in Fortran, Java or other 
languages, such as Python. As long as no pointers or derive classes are 
used the format should be fairly simple to re-implement reading and 
writing such files in another language.

> To send data to "other" applications, there is always the XML archive.

For new applications we are already defining XML applications to 
exchange data, but large data sets will always be stored in a binary 
format and referenced from these XML files.

Matthias