[Boost-users] [interprocess] Reading huge files

Subject: [Boost-users] [interprocess] Reading huge files
From: Sensei (senseiwa_at_[hidden])
Date: 2013-10-11 08:55:07

Dear all,

I am new to boost memory mapping, so this question might look simplistic.

I need to read huge amounts of data (for instance, a 20GB file), and 
since memory mapping is quite fast, I was going to use it. However, I 
don't know what it would be faster when, due to memory constraints, I 
need to partition the file into regions. Moreover, I should treat the 
file as a string (I need to perform string operations).

What I'm trying now is just to read the entire file:

     boost::interprocess::file_mapping mmap(input_filename.c_str(), 
boost::interprocess::read_only);
     boost::interprocess::mapped_region map(mmap, 
boost::interprocess::read_only);

     std::size_t l = map.get_size(), tot_read = 0;

     void *ptr = map.get_address();

     while (tot_read < l)
     {
         register std::size_t x = std::min(l - tot_read, 
static_cast<std::size_t>(prealloc));

         std::copy_n(static_cast<char*>(ptr) + tot_read, x, line.begin());

         // Do something here...

         tot_read += x;
     }

So, when the file is huge, do I need to create a mapped_region inside 
the loop? I didn't see anywhere in the documentation the possibility to 
move the mapped region.

Another side-question, if you don't mind. I'm not sure that what I'm 
doing is efficient, especially the need to copy from the region to a 
string. If you have suggestions, I'm more than happy to hear these.

Thanks & Cheers!