[Boost-users] Serialization cumulatively.

Subject: [Boost-users] Serialization cumulatively.
From: Tony Camuso (tcamuso_at_[hidden])
Date: 2015-03-12 09:09:17

I'm trying to serialize the data from multiple passes of an app
on various files. Each pass generates data that must be serialized.
If I simply append each serialization, then deserialization will
only occur for the first instance, since the number of records
read by the deserialization will only be for the first instance.

What I'm doing is deserializing on each new pass, deleting the
original file, and then serializing everything again with the
new information.

If there were only a few files to process, this would not be a
problem. However there are thousands of files.

Additionally, on each new pass, I am checking to see if a
certain type of record has already been saved. So, with every
pass, I must look up in a deeper and deeper database.

Currently, it's taking almost an hour to process about 3000
files, with an average of 55,000 lines per file. It is a
huge amount of data.

However, I'm looking for a way to reduce the length of time it
takes to do this processing.

Does anybody have a better idea than to cycle through the
serialize-deserialize-lookup-serialize sequence for each
file?