[Boost-bugs] [Boost C++ Libraries] #4374: Shared Memory Bus Error

Subject: [Boost-bugs] [Boost C++ Libraries] #4374: Shared Memory Bus Error
From: Boost C++ Libraries (noreply_at_[hidden])
Date: 2010-06-24 14:35:38

#4374: Shared Memory Bus Error
--------------------------------------------+-------------------------------
 Reporter:  Matt Cupp <cuppmatt@…>          |       Owner:  igaztanaga  
     Type:  Bugs                            |      Status:  new         
Milestone:  Boost 1.44.0                    |   Component:  interprocess
  Version:  Boost 1.42.0                    |    Severity:  Problem     
 Keywords:  shm ipc tmpfs                   |  
--------------------------------------------+-------------------------------
 Found on StackOverflow at http://stackoverflow.com/questions/3103255

 /boostinterprocess-shared-memory-bus-error

 I'm using CentOS 5.4 x86_64 and Boost 1.42.0 on a cluster that uses Open-
 MPI 1.3.3. I'm writing a shared library that uses shared memory to store
 large amounts of data for multiple processes to use. There's also a loader
 application that will read in the data from the files and load them into
 the shared memory.

 When I run the loader application, it determines the amount of memory that
 it needs to store the data exactly then adds 25% for overhead. For just
 about every file, it'll be over 2 gigs worth of data. When I make the
 memory request using Boost's Interprocess library, it says it has
 successfully reserved the requested amount of memory. But when I use start
 to use it, I get a "Bus error". From what I can tell, the bus error is a
 result of accessing memory outside the range that is available for the
 memory segment.

 So I started looking into how the shared memory is on Linux and what to
 check to make sure my system is correctly configured to allow that large
 amount of shared memory.

    1. I looked at the "files" at /proc/sys/kernel/shm*:
           * shmall - 4294967296 (4 Gb)
           * shmmax - 68719476736 (68 Gb)
           * shmmni - 4096

    2. I called the ipcs -lm command:

       ------ Shared Memory Limits --------
       max number of segments = 4096
       max seg size (kbytes) = 67108864
       max total shared memory (kbytes) = 17179869184
       min seg size (bytes) = 1

 From what I can tell, those settings indicate that I should be able to
 allocate enough shared memory for my purposes. So I created a stripped
 down program that created large amounts of data in shared memory and
 compiled it with the line:

 g++ ShmTest.cpp -lboost_system -lrt

 Then ran it and attached output (edited to make it smaller).

 If you run the program with the a parameter (any will work, just need to
 increase the argc), it preallocate the vector but will still result in a
 bus error at the same array index.

 I checked the size of the "files" at /dev/shm using the ls -ash /dev/shm
 command:

 total 2.0G
    0 .     0 ..  2.0G ShmTest

 And just like with my original application it the size of the allocated
 shared memory is capped at 2 gigs. Given that it "successfully" allocated
 2352000000 bytes of memory, in gigabytes (using 1024*1024*1024) it should
 be 2.19 Gb.

 On Linux the shared memory mechanisms it uses (tmpfs) by default limits it
 to half the system RAM. So on the my cluster it’s 2 Gb because we have 4
 Gb system RAM. So when it tried to allocate the shared memory segment, it
 allocated up to the max size left on the /dev/shm.

 But issue came when the Boost library didn’t indicate an error or even the
 report correct amount of free memory when it couldn’t allocate the
 requested amount of memory. It was just happy to apparently chug along
 until it reached the end of the segment and then errored.

 The long term solution is to update the /etc/fstab file to make the change
 permanently, but a command line call can be run to increase the size of
 the available shared memory on each node until reboot.

 mount -o remount,size=XXX /dev/shm

 Where the XXX is the amount of memory to make available (for example
 size=4G).

 Shouldn't Boost have a way to detect that it doesn't actually have the
 full amount available?  Instead of blindly letting the user allocate
 memory and then error?

-- 
Ticket URL: <https://svn.boost.org/trac/boost/ticket/4374>
Boost C++ Libraries <http://www.boost.org/>
Boost provides free peer-reviewed portable C++ source libraries.