Re: [Boost-users] serialization & several text archive within the samefiles...

From: François Mauger (mauger_at_[hidden])
Date: 2008-03-24 05:12:26

Hi robert and all,

>>>On Sun, March 23, 2008 10:11 pm, Robert Ramey wrote:
> if you don't like tracking - you can turn it off for the types you want.

see below.

> to add your own characters between archives try something like this:
>

this is what I have done of course, to make it run.
but I was considering that such behavior is the responsability
of the core lib, not the user (me), that's my point.
As I use a 'serialization manager' class that implements
some txt/xml/bin archive depending of the file extension
provided  by the users of my lib, I would find more, say, 'elegant'
not to treat text archives in a different way considering xml/bin ars.
but maybe it is a purist issue! and maybe not really relevant.
of course, this is not a pb if I add this 'blank' char by hand.
So it is fine for me.

-- next point:

About memory tracking: after a few investigations while
considering my needs, I find it very powerful.
I only implies some care while using it.
As I use serialization by pointers on nested objects
in my lib, tracking is very useful for it handle the links
properly.

The only problem I met is the following:
- I have say 1000000 records (some instances of a class) to write
in the archive.
- Each record uses massively std::vector or std::list as internal members
  (subrecords with dynamically allocated memory)
- Each record (with its internal subrecords)
  is more than 1000 bytes long --> so total storage for my data set
  is about 1Gb in a single output file!
- More, each record has pointers members in it, so tracking is activated
  and I NEED it.

Now, when I write all the records using a kind of loop on my archive,
I must not reuse the same memory addresses (as explained in the sample
programs and docs in boost::serialization, I cannot use a temporary
record instance in the loop).
So I have to store in the RAM of the machine
the whole sets of records (and dynamically allocated subrecords)
at the same time before to store them in the archive (1Gb!).
Typically I use a std::list for this.
It implies to run a machine with 1Gb available RAM,
which cannot be garanted for all my users,
even on our computing center...

On the other side, during the loop process,
there is no way to erase (nor reduce the size of)
previous written records while saving the current one
in the archive. Of course this would
save the current running memory but this available memory
could be 'reused' at some point by the
tracking mechanism (this is rather a random process that depends on the
system)
and then one would experience some misinterpretation of data
as already serialized stuff through my pointers. Then
the output will be corrupted.

This is what I call a "long-range memory tracking effect":

- within a single record (short range),
memory tracking is fine for it enables to
maintain some connections between 'subrecords' through pointers
in a very nice and 'storage saving' way.
More de-serialization works perfectly, without duplicate
records/subrecords and memory leak issues.

- for successive records (long range),
I have no need of pointers to make links
between objects (records and subrecords in it),
so tracking is unuseful but it is still activated
in the same program!
Then it leads to some nesting between memory addresses among
different records. This is a corruption case, unless, as I explained
above, all records (and internal subrecords) are kept in memory
till the end of the serialization process... then I need 1 Gb RAM
machine!

The only way I have found in my progs to break this long range effect
is to use one archive per record. It then seems that
memory tracking is confined within the limit of the current archive
which is exactly what I want.

Finally, what will be useful to enable my "per-record" serialization
approach for a large data set, this is a kind of "memory tracking reset"
function that could be invoked online while looping on the archive.
I have no idea if it is possible to implement this, and if other
people could find it useful, Robert first!

Hope my point has been understood.
At least, if you can confirm that tracking mechanism is confined
within a single archive, I can use this strategy of
multiple archives per file.

Thanks a lot for your attention, advice and constructive critics.

And many many thanks to Robert
for this very nice and elegant serialization
library. This is really a great and useful work!

frc
--
> François Mauger wrote:
>
>> Hi
>>
>>
>> I have large amount of data to store/load from files.
>> I use boost::serialization library to do it.
>> It has very nice features.
>>
>>
>> I use version 1.33:
>>
>>>>> libboost-serialization-dev                 1.33.1-9ubuntu3.1
>>
>> Checking different strategies to play with my data and i/o archives,
>> I met the following problem:
>> if I save several text_oarchives within the same output file (a trick to
>> break side-effects of memory tracking), then the deserialization failed
>> for there is no separator between successive text archives.
>>
>> I have to explicitely add a 'std::endl' in the output stream
>> to make it run.
>>
>> This pb does not appear
>> with xml archives for the </tag> at the end is unambiguous to parse the
>> end of each archive. I did not check for binary archives, but I guess
>> there will be no pb.
>>
>> For me a mandatory 'white' character
>> should be added as the last byte in a text output archive (when
>> destructor is invoked?). This will make more coherent (symmetric!) in
>> comparison with xml/binary ars. This is only a suggestion: I cannot
>> imagine all the side-effects such a strategy could imply.
>>
>> A sample demo file is attached.
>>
>>
>> Thanks for your attention.
>>
>>
>> frc
>>
>>> _______________________________________________
>>> Boost-users mailing list
>>> Boost-users_at_[hidden]
>>> http://lists.boost.org/mailman/listinfo.cgi/boost-users
>>>
>
>
>
> _______________________________________________
> Boost-users mailing list
> Boost-users_at_[hidden]
> http://lists.boost.org/mailman/listinfo.cgi/boost-users
-- 
Francois Mauger
Laboratoire de Physique Corpusculaire de Caen et Universite de Caen
ENSICAEN - 6, Boulevard du Marechal Juin, 14050 CAEN Cedex, FRANCE
e-mail: mauger_at_[hidden]
tel.: (0/[phone]
fax: (0/[phone]