Re: [boost] Boost Serialization: Serializing large objects

Subject: Re: [boost] Boost Serialization: Serializing large objects
From: Adlai Shawareb (adlai_at_[hidden])
Date: 2017-10-11 21:28:32

> You say the object is 700 megs. Is that 700 megs in memory or in text? The text representation is almost certainly bigger than the in memory
> representation so you might just be running out of memory.
>
> Alternatively, you could just be running out of address space if stringstream requires a contiguous buffer. That's an implementation detail, but I
> would not be surprised if stringstream allocates a contiguous buffer, in which case you almost certainly would be bumping into a memory
> fragmentation issue due to the 32bit address space.
>
> If you serialize directly to a file instead of stringstream, as Robert suggested, the problem might go away.
>
> -- chris

Thanks for responding Chris. The unserialized data is 700 megs. You are correct, the text representation is closer to 1.2 GB. We are using extended addressing on QNX, so I haven't focused on the 32bit address space / fragmentation issue, but it can't be ruled out. The contiguous buffer issue seems likely. The questions I was asking of Robert were to look for something that we could take to QNX to get some feedback.

Adlai

________________________________