Re: [boost] Re: Serialization (Persistence) library draft submission

From: Matthias Troyer (troyer_at_[hidden])
Date: 2002-02-28 01:47:08

On Thursday, February 28, 2002, at 12:29 AM, rameysb wrote:

> --- In boost_at_y..., Matthias Troyer <troyer_at_i...> wrote:
>>
>> On Wednesday, February 27, 2002, at 10:48  PM, rameysb wrote:
>>>
>>>
>>> also the rational for using text for numbers - is portability from
>>> machine to machine.  binary portability would be a huge hassle and
>>> never be complete.  Its bad enough that there are little and big
>>> endien machines, floating point formats vary from machine to
>>> machine.  I believe that the only practical way to make archives
>>> portable across machines to use text files.
>>>
>>> Robert Ramey
>>>
>>
>> No, there are other formats like XDR that can be used. My
>> files for storage of a "checkpoint" are often >100 MB in
>> size, I can't grow that to 3-4 times the size just because
>> a library requires text.
>>
>> Any serialization library should allow various data formats,
>> binary or text, otherwise it will be of very limited use.
>>
>> Matthias
>
> How about this.  We'll add a an archive
>
>     oarchive & oarchive.binary_write(size_t length, void
> *data_address);
>
> and a corresponding loading operator
>
>     iarchive & iarchive.binary_read(size_t &length, void
> *data_address);
>
> so that the save/load functions can write/read binary data
>
> e.g.
>
> void T::save(oarchive &ar) const
> {
>     ar << data_length;
>     ar.binary_write(data_length, data_address);
> }
>
> void T::load(iarchive &ar, int file_version)
> {
>     ar >> data_length;
>     // check that data length matches the structure size
>     // or allocate a new object - or what ever
>     ar.binary_read(data_length, &data);
> }
>
> This is all that would be required to support binary objects an an
> arbitrary size.  The only aspect of the system that would be
> conpromised would be portability from environment to environment.
> Presumably, this is not a consideration for an application that wants
> to do this.

This will not work either for just the reason you want to use text
formats for: if protability is desired some conversion, based on the
type, has to be done and you cannot just use the native binary
format.

Furthermore this means adding a SECOND save and load function,
one for text and one for binary serialization, or one for each
binary format.

> As to file sizes, I believe that that should be approached from a
> more general perspective.  There should be a stream and /or streambuf
> which implements data compression.  Since this library uses any
> stream passed to it, it could utilize compression implemented in this
> way.
>
> That is the issue of file size ( or equivalently, elmination of
> redundancy) could and should be addressed in a more general way
> somewhere else.  If this were done, one could have full binary
> portability and minimum file size using text i/o

I don't think compressing text I/O is as efficient as directly storing
binary data.