Subject: Re: [boost] Boost Serialization: Serializing large objects
From: Adlai Shawareb (adlai_at_[hidden])
Date: 2017-10-11 21:28:32
> You say the object is 700 megs. Is that 700 megs in memory or in text? The text representation is almost certainly bigger than the in memory
> representation so you might just be running out of memory.
> Alternatively, you could just be running out of address space if stringstream requires a contiguous buffer. That's an implementation detail, but I
> would not be surprised if stringstream allocates a contiguous buffer, in which case you almost certainly would be bumping into a memory
> fragmentation issue due to the 32bit address space.
> If you serialize directly to a file instead of stringstream, as Robert suggested, the problem might go away.
> -- chris
Thanks for responding Chris. The unserialized data is 700 megs. You are correct, the text representation is closer to 1.2 GB. We are using extended addressing on QNX, so I haven't focused on the 32bit address space / fragmentation issue, but it can't be ruled out. The contiguous buffer issue seems likely. The questions I was asking of Robert were to look for something that we could take to QNX to get some feedback.
Boost list run by bdawes at acm.org, gregod at cs.rpi.edu, cpdaniel at pacbell.net, john at johnmaddock.co.uk