Re: [Boost-users] Serialization cumulatively.

Subject: Re: [Boost-users] Serialization cumulatively.
From: tcamuso (tcamuso_at_[hidden])
Date: 2015-03-20 10:37:09

Hi, Robert.

I would have answered sooner, but had other issues arise.

I had a look at your code, and that's basically what I'm already doing. 

Problem is that the time to process the files this way is O(log n), as
processing each file takes incrementally longer as the database grows. It
takes about an hour to process around 3000 files having about 15GB of data.
Sounds reasonable, until you compare it to the compiler that whizzes through
all the same files, and more, in only a few minutes.

When I serialize the output without trying to recreate the whole database
for each file, the length of time to process these 3000 files drops to about
5 minutes, which is a much more acceptable number for my target users. This
yields one very large file with about 3000 appended serializations. 

What I'd like to do, because i think it would be much faster, is to go
through the one big file and deserialize each of those serializations as
they are encountered. Early testing showed that it would only take a few
minutes to integrate these pieces into one whole.

If there were linefeeds in the serialized data, the code to do this would be
much simpler. 

Is there another, more architected way for me to deserialize an aggregate of
serializations?

--
View this message in context: http://boost.2283326.n4.nabble.com/Serialization-cumulatively-tp4673059p4673531.html
Sent from the Boost - Users mailing list archive at Nabble.com.