Boost logo

Boost Users :

Subject: [Boost-users] Question regarding boost binary serializer which is eating 90GB of memory, while re-reading the same content.
From: dinesh kumar (dineshkumar02_at_[hidden])
Date: 2018-01-20 07:52:11

Hi All,

Greetings for the day.

I am trying to understand the behavior of Boost's binary serializer(Boost
version 1.65). Please consider the below example, where I am inserting one
record into the file, and trying to read the same record 5 times.

#include <boost/archive/binary_iarchive.hpp>
#include <boost/archive/binary_oarchive.hpp>
#include <boost/serialization/string.hpp>
#include <fstream>
#include <iostream>

using namespace std;
using namespace boost::archive;
class logEntry {
size_t m_txID;
string m_jsonStr;

friend class boost::serialization::access;
template <typename Archive>
friend void serialize( Archive &ar, logEntry &l, const unsigned int version

logEntry() {
m_txID = 0;
m_jsonStr = "";
logEntry( size_t id, const string &val ) {
m_txID = id;
m_jsonStr = val;
string getJsonValue() {
return m_jsonStr;

size_t getTxId() {
return m_txID;

template <typename Archive>
void serialize( Archive &ar, logEntry &l, const unsigned int version ) {
ar &l.m_txID;
ar &l.m_jsonStr;

size_t prevReadPos = 0;

void save( size_t n ) {
ofstream file{"/tmp/test.bin", ios::binary | ios::trunc};
binary_oarchive oa{file};

// Save n records
for ( int i = 0; i < n; i++ )
oa << logEntry( i, "{Some Json String}" );


// Load data batch wise
void load( size_t bsize ) {
ifstream file{"/tmp/test.bin", ios::binary};
binary_iarchive ia{file};

// Record file length
size_t fileEnd;
size_t beg = file.tellg();

file.seekg( 0, ios::end );
fileEnd = file.tellg();
file.seekg( beg, ios::beg );

logEntry l;
for ( size_t i = 0; i < bsize; i++ ) {
* ia >> l;*
* // Reset the file read position to the beginning of the file to read the
same line one more time*
* file.seekg( beg, ios::beg );*

prevReadPos = file.tellg();

int main() {
// Saving only one record
save( 1 );
while ( 1 ) {
// Trying to read the same record 5 times
load( 5 );
sleep( 5 );

>From the above code, I am able to read the record one time and whenever I
re-set the file pointer back to the beginning of the file, the program
"Crashes with SIGKILL".
And also, I observed that this process it consumed around *90GB *of
memory(I got this reading from xCode while debugging the code).

My dev environment is as below.

clang --version
Apple LLVM version 8.0.0 (clang-800.0.42.1)
Target: x86_64-apple-darwin15.6.0
Thread model: posix

Kindly let me know if you need any more information from my side.


Boost-users list run by williamkempf at, kalb at, bjorn.karlsson at, gregod at, wekempf at