Subject: [Boost-bugs] [Boost C++ Libraries] #4374: Shared Memory Bus Error
From: Boost C++ Libraries (noreply_at_[hidden])
Date: 2010-06-24 14:35:38
#4374: Shared Memory Bus Error
--------------------------------------------+-------------------------------
Reporter: Matt Cupp <cuppmatt@â¦> | Owner: igaztanaga
Type: Bugs | Status: new
Milestone: Boost 1.44.0 | Component: interprocess
Version: Boost 1.42.0 | Severity: Problem
Keywords: shm ipc tmpfs |
--------------------------------------------+-------------------------------
Found on StackOverflow at http://stackoverflow.com/questions/3103255
/boostinterprocess-shared-memory-bus-error
I'm using CentOS 5.4 x86_64 and Boost 1.42.0 on a cluster that uses Open-
MPI 1.3.3. I'm writing a shared library that uses shared memory to store
large amounts of data for multiple processes to use. There's also a loader
application that will read in the data from the files and load them into
the shared memory.
When I run the loader application, it determines the amount of memory that
it needs to store the data exactly then adds 25% for overhead. For just
about every file, it'll be over 2 gigs worth of data. When I make the
memory request using Boost's Interprocess library, it says it has
successfully reserved the requested amount of memory. But when I use start
to use it, I get a "Bus error". From what I can tell, the bus error is a
result of accessing memory outside the range that is available for the
memory segment.
So I started looking into how the shared memory is on Linux and what to
check to make sure my system is correctly configured to allow that large
amount of shared memory.
1. I looked at the "files" at /proc/sys/kernel/shm*:
* shmall - 4294967296 (4 Gb)
* shmmax - 68719476736 (68 Gb)
* shmmni - 4096
2. I called the ipcs -lm command:
------ Shared Memory Limits --------
max number of segments = 4096
max seg size (kbytes) = 67108864
max total shared memory (kbytes) = 17179869184
min seg size (bytes) = 1
From what I can tell, those settings indicate that I should be able to
allocate enough shared memory for my purposes. So I created a stripped
down program that created large amounts of data in shared memory and
compiled it with the line:
g++ ShmTest.cpp -lboost_system -lrt
Then ran it and attached output (edited to make it smaller).
If you run the program with the a parameter (any will work, just need to
increase the argc), it preallocate the vector but will still result in a
bus error at the same array index.
I checked the size of the "files" at /dev/shm using the ls -ash /dev/shm
command:
total 2.0G
0 . 0 .. 2.0G ShmTest
And just like with my original application it the size of the allocated
shared memory is capped at 2 gigs. Given that it "successfully" allocated
2352000000 bytes of memory, in gigabytes (using 1024*1024*1024) it should
be 2.19 Gb.
On Linux the shared memory mechanisms it uses (tmpfs) by default limits it
to half the system RAM. So on the my cluster itâs 2 Gb because we have 4
Gb system RAM. So when it tried to allocate the shared memory segment, it
allocated up to the max size left on the /dev/shm.
But issue came when the Boost library didnât indicate an error or even the
report correct amount of free memory when it couldnât allocate the
requested amount of memory. It was just happy to apparently chug along
until it reached the end of the segment and then errored.
The long term solution is to update the /etc/fstab file to make the change
permanently, but a command line call can be run to increase the size of
the available shared memory on each node until reboot.
mount -o remount,size=XXX /dev/shm
Where the XXX is the amount of memory to make available (for example
size=4G).
Shouldn't Boost have a way to detect that it doesn't actually have the
full amount available? Instead of blindly letting the user allocate
memory and then error?
-- Ticket URL: <https://svn.boost.org/trac/boost/ticket/4374> Boost C++ Libraries <http://www.boost.org/> Boost provides free peer-reviewed portable C++ source libraries.
This archive was generated by hypermail 2.1.7 : 2017-02-16 18:50:03 UTC