Re: [Boost-users] Serialization: hugo file size, binary archives

From: Robert Ramey (ramey_at_[hidden])
Date: 2006-04-04 23:01:32

There is no information written into a binary archive which is not
absolutly necessary.  That is there is not redundant information.
If you archives are "too" big there must be some mistake.  I would
suggest that you output (part of) the archive using text or xml format
so you can see what is actually being written and how it differs
from what you expect.

Robert Ramey

Sascha Ochsenknecht wrote:
> Hello,
>
> First of all, thanks for the quick reply.
>
> I tried std::vector resize(), but the problem still exist. I also
> removed some redundant data from my data structure (its data which can
> be generated by a postprocess after reading the archive). I still have
> about 1.0GByte.
>
> Is there somewere a doc were 'overhead' data is documented?
> Would it be helpful if I send you a generated archive (with other
> data I can generate uncompressed archives around 1 MByte)? Would
> somebody have a look on it.
>
> Another thing ... I get a lot of compile warnings about unused
> variables within the serialization library. Would be nice if these
> can be fixed with the next release.
>
> Thanks in advance.
>
> Sascha
>
> Robert Ramey wrote:
>> I'm am not aware of serialization causing such a problem.
>> You might investigate std::vector resize(), etc to see if
>> the vector really has a lot of null data.
>>
>> Robert Ramey
>>
>> Sascha Ochsenknecht wrote:
>>> Hello,
>>>
>>> I'm using the Serialization Library of Boost to store my data
>>>    structure. I want to use the binary archive type by default:
>>> boost::archive::binary_oarchive(ostream &s) // saving
>>> boost::archive::binary_iarchive(istream &s) // loading
>>>
>>> But I noticed that these files can be very big compared to the
>>> stored data. I got a binary archive with around 1.5GByte. That
>>> could be but when I compress it I got only ~200MByte left (!).
>>> It seems that there is a lot of 'overhead' data or 'redundant' data
>>> (I see a lot of '0' when I look into it with an Hex editor).
>>>
>>> i tried the gzip (...) filter of the Iostreams library, but I want
>>> to avoid this for production due to increasing runtime.
>>>
>>> Some Information about my data structure (maybe helpful):
>>> - using a lot of pointer
>>> - using a lot of std::vector
>>>
>>> Does anybody investigate the same problem?
>>> Is there a possibility to decrease the archive size but storing the
>>> same amount of data?
>>> What could be a solution? Writing an own/optimized (regarding to my
>>> data structure) Archive class?
>>>
>>> thanks in advance
>>> Sascha