Re:[boost] Serializer and when to stop

From: Robert Ramey (ramey_at_[hidden])
Date: 2004-06-05 23:14:49

Darren Cook wrote:

> I'm using boost::serializer between two programs: the first creates the
> data then serializes it, the second loads that data in and analyzes it.

> When I had just 420 data samples it all worked fine: I create a
> vector<Sample> in memory, serialized it and loaded it in the other side.
> When I moved to 7000 data samples the first program, the creator, sucked
> all the memory from the machine and then some.

7000 samples? That doesn't seem very large in today's environment. I can't
see why creating an archive should consume any significant memory at all.
Nothing significant is constructed.  Perhaps is a bug somewhere.

In any case, you do raise in interesting issue which would crop up if
serialization were used in something like data logging.

> So I moved to serializing each data sample as it was created. 
> To read it back in I changed my code from this:

> std::ifstream input_file(fname,std::ios::binary);
> boost::archive::binary_iarchive archive(input_file); 
> std::vector<Sample> samples;
> archive>>samples;
> input_file.close();

> Into this:

> std::ifstream input_file(fname,std::ios::binary);
> boost::archive::binary_iarchive archive(input_file);

> std::vector<Sample> samples;
> samples.reserve(8000);

> while(!input_file.eof()){
>	Sample s;
>	archive>>s;
>	samples.push_back(s);
>	}

> input_file.close();

> But every time it fails with an assert [1]. This seems to be when it has
>  reached end of file. I cannot know in advance how many data samples I'll
>  write to disk, so it seems I have three options:
>    A: Make a special "zero" version of Sample to mark end of file
>    B: At end of program 1 write the number of samples to a special file, 
>  and read that in before starting the above loop, so I know when to stop.
>    C: Alter the serializer class to throw exception instead of an assert;
>  I can then catch it and carry on.

I think I changed the assert/exception to check before loading rather than
after for the next version.  This would permit your code above to work as
you expect. I'll have to double check.

> What is the best way? Is adding a terminator byte to boost::serializer an
> option (i.e. so I don't have to make a terminator version of each data 
> structure I want to serialize).

> Incidentally my above loop does a copy of each sample as it adds it to
>  the vector. Is the above code going to be common enough to make it worth
>  including into boost::serializer as a helper function, which could then
>  be optimized with in-place construction or something clever like that?
> (And then it could handle the termination handling itself as well.)

The default implementation of STL collections create a new element on the
stack and copy into the collection.  Its as clever as I can make it without
getting tripped up in things like exception safety, objects without default
constructors, etc.

Robert Ramey