[Boost-users] Question regarding boost binary serializer which is eating 90GB of memory, while re-reading the same content.

Subject: [Boost-users] Question regarding boost binary serializer which is eating 90GB of memory, while re-reading the same content.
From: dinesh kumar (dineshkumar02_at_[hidden])
Date: 2018-01-20 07:52:11

Hi All,

Greetings for the day.

I am trying to understand the behavior of Boost's binary serializer(Boost
version 1.65). Please consider the below example, where I am inserting one
record into the file, and trying to read the same record 5 times.

#include <boost/archive/binary_iarchive.hpp>
#include <boost/archive/binary_oarchive.hpp>
#include <boost/serialization/string.hpp>
#include <fstream>
#include <iostream>

using namespace std;
using namespace boost::archive;
class logEntry {
 private:
size_t m_txID;
string m_jsonStr;

friend class boost::serialization::access;
template <typename Archive>
friend void serialize( Archive &ar, logEntry &l, const unsigned int version
);

 public:
logEntry() {
m_txID = 0;
m_jsonStr = "";
}
logEntry( size_t id, const string &val ) {
m_txID = id;
m_jsonStr = val;
}
string getJsonValue() {
return m_jsonStr;
}

size_t getTxId() {
return m_txID;
}
};

template <typename Archive>
void serialize( Archive &ar, logEntry &l, const unsigned int version ) {
ar &l.m_txID;
ar &l.m_jsonStr;
}

size_t prevReadPos = 0;

void save( size_t n ) {
ofstream        file{"/tmp/test.bin", ios::binary | ios::trunc};
binary_oarchive oa{file};

// Save n records
for ( int i = 0; i < n; i++ )
oa << logEntry( i, "{Some Json String}" );

file.flush();
file.close();
}

// Load data batch wise
void load( size_t bsize ) {
ifstream        file{"/tmp/test.bin", ios::binary};
binary_iarchive ia{file};

// Record file length
size_t fileEnd;
size_t beg = file.tellg();

file.seekg( 0, ios::end );
fileEnd = file.tellg();
file.seekg( beg, ios::beg );

logEntry l;
for ( size_t i = 0; i < bsize; i++ ) {
* ia >> l;*
* // Reset the file read position to the beginning of the file to read the
same line one more time*
* file.seekg( beg, ios::beg );*
}

prevReadPos = file.tellg();
file.close();
}

int main() {
// Saving only one record
save( 1 );
while ( 1 ) {
// Trying to read the same record 5 times
load( 5 );
sleep( 5 );
}
}

>From the above code, I am able to read the record one time and whenever I
re-set the file pointer back to the beginning of the file, the program
"Crashes with SIGKILL".
And also, I observed that this process it consumed around *90GB *of
memory(I got this reading from xCode while debugging the code).

My dev environment is as below.

clang --version
Apple LLVM version 8.0.0 (clang-800.0.42.1)
Target: x86_64-apple-darwin15.6.0
Thread model: posix
InstalledDir:
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

Kindly let me know if you need any more information from my side.
-- 
Regards,
Dinesh
manojadinesh.blogspot.com