[interprocess] How to know how much memory is taken up by an object allocated in boost interprocess shared memory?
Re-posting, as the boost mailing list rejected the first post <http://boost.2283326.n4.nabble.com/How-to-know-how-much-memory-is-taken-up-by-an-object-allocated-in-boost-interprocess-shared-memory-tc4680682.html> mentioning that I had to add "[interprocess]" to the subject line. I have asked this question on stackoverflow <http://stackoverflow.com/questions/30642585/why-does-an-object-allocated-in-boost-interprocess-shared-memory-take-up-more-me> , but got no response. For this program: #include <iostream> #include <boost/interprocess/mapped_region.hpp> #include <boost/interprocess/managed_shared_memory.hpp> #include <boost/interprocess/containers/list.hpp> #include <iostream> #define SHARED_MEMORY_NAME "SO12439099-MySharedMemory" #define DATAOUTPUT "OutputFromObject" #define INITIAL_MEM 650000 #define STATE_MATRIX_SIZE 4 using namespace std; namespace bip = boost::interprocess; class SharedObject { public: unsigned int tNumber; bool pRcvdFlag; bool sRcvdFlag; unsigned long lTimeStamp; }; typedef bip::allocator<SharedObject, bip::managed_shared_memory::segment_manager> ShmemAllocator; typedef bip::list<SharedObject, ShmemAllocator> SharedMemData; int main() { bip::managed_shared_memory* seg; SharedMemData *sharedMemOutputList; bip::shared_memory_object::remove(DATAOUTPUT); seg = new bip::managed_shared_memory(bip::create_only, DATAOUTPUT, INITIAL_MEM); const ShmemAllocator alloc_inst(seg->get_segment_manager()); sharedMemOutputList = seg->construct<SharedMemData>("TrackOutput")(alloc_inst); std::size_t beforeAllocation = seg->get_free_memory(); std::cout<<"\nBefore allocation = "<< beforeAllocation <<"\n"; SharedObject temp; sharedMemOutputList->push_back(temp); std::size_t afterAllocation = seg->get_free_memory(); std::cout<<"After allocation = "<< afterAllocation <<"\n"; std::cout<<"Difference = "<< beforeAllocation - afterAllocation <<"\n"; std::cout<<"Size of SharedObject = "<< sizeof(SharedObject) <<"\n"; std::cout<<"Size of SharedObject's temp instance = "<< sizeof(temp) <<"\n"; }//main The output is: Before allocation = 649680 After allocation = 649632 Difference = 48 Size of SharedObject = 16 Size of SharedObject's temp instance = 16 If the size of SharedObject and it's instance is 16 bytes, then how can the difference in allocation be 48? Even if padding had automatically been done, it's still too much to account for 3 times the size (for larger structures it goes to 1.33 times the size). Because of this, I'm unable to allocate and dynamically grow the shared memory reliably. If SharedObject contains a list which grows dynamically, that could add to the uncertainty of space allocation even more. How can these situations be safely handled? How can I know exactly how much of memory to allocate? -- View this message in context: http://boost.2283326.n4.nabble.com/interprocess-How-to-know-how-much-memory-... Sent from the Boost - Dev mailing list archive at Nabble.com.
On 08/10/2015 5:30, nav999 wrote:
The output is:
Before allocation = 649680 After allocation = 649632 Difference = 48 Size of SharedObject = 16 Size of SharedObject's temp instance = 16
If the size of SharedObject and it's instance is 16 bytes, then how can the difference in allocation be 48? Even if padding had automatically been done, it's still too much to account for 3 times the size (for larger structures it goes to 1.33 times the size). Because of this, I'm unable to allocate and dynamically grow the shared memory reliably. If SharedObject contains a list which grows dynamically, that could add to the uncertainty of space allocation even more.
How can these situations be safely handled? How can I know exactly how much of memory to allocate?
Remember that any allocation, even with new or malloc, has a payload, in order to store information about how to deallocate that memory, how to merge that buffer with adjacent buffers, etc.. This happens with your system malloc (typically, 8-16 extra bytes per allocation plus extra alignment) The memory allocator in shared memory has an overhead of 4-8 bytes (in 32 bits systems, 8-16 in 64 bit systems) Then you need to store the number of objects in order to call destructors (you an allocate arrays) and you've done a named allocation. You need to store the string and metadata (a pointer to the string and maybe that this was a "named allocation" and not an anonymous or "instance allocation) in shared memory. So 16 bytes of data + 8 bytes from the memory allocator + 8 bytes to store the pointer to the name plus metadata plus 12 bytes from the string "TrackOutput" (including null-end) plus rounding, you get 48 bytes. The overhead is constant (except the name you supply) for each allocation. So the constant factor 1.33 is not for all objects only for small ones. ¿How do you know how many heap you need to allocate elements in RAM? It's exactly the same problem (you allocate until malloc says there is no more memory). You must also note that memory gets fragmented with allocations and reallocations, so even there is free memory, it maybe could not service your request because there is no contiguous memory big enough to fulfill your request. A shared memory can't be automatically expanded so fragmentation is a much bigger issue than with heap memory. Ion
On 08/10/2015 5:30, nav999 wrote:
The output is:
Before allocation = 649680 After allocation = 649632 Difference = 48 Size of SharedObject = 16 Size of SharedObject's temp instance = 16
If the size of SharedObject and it's instance is 16 bytes, then how can the difference in allocation be 48? Even if padding had automatically been done, it's still too much to account for 3 times the size (for larger structures it goes to 1.33 times the size). Because of this, I'm unable to allocate and dynamically grow the shared memory reliably. If SharedObject contains a list which grows dynamically, that could add to the uncertainty of space allocation even more.
How can these situations be safely handled? How can I know exactly how much of memory to allocate? ... [show rest of quote]
Remember that any allocation, even with new or malloc, has a payload, in order to store information about how to deallocate that memory, how to merge that buffer with adjacent buffers, etc.. This happens with your system malloc (typically, 8-16 extra bytes per allocation plus extra alignment)
The memory allocator in shared memory has an overhead of 4-8 bytes (in 32 bits systems, 8-16 in 64 bit systems)
Then you need to store the number of objects in order to call destructors (you an allocate arrays) and you've done a named allocation. You need to store the string and metadata (a pointer to the string and maybe that this was a "named allocation" and not an anonymous or "instance allocation) in shared memory.
So 16 bytes of data + 8 bytes from the memory allocator + 8 bytes to store the pointer to the name plus metadata plus 12 bytes from the string "TrackOutput" (including null-end) plus rounding, you get 48 bytes.
The overhead is constant (except the name you supply) for each allocation. So the constant factor 1.33 is not for all objects only for small ones.
¿How do you know how many heap you need to allocate elements in RAM? It's exactly the same problem (you allocate until malloc says there is no more memory).
You must also note that memory gets fragmented with allocations and reallocations, so even there is free memory, it maybe could not service your request because there is no contiguous memory big enough to fulfill your request. A shared memory can't be automatically expanded so fragmentation is a much bigger issue than with heap memory.
Ion
Thank you Ion. I agree with pre-allocating memory, but even in such a case, it helps to know how much of padding to allocate. This is what I got when I tabulated values for multiple datastructure sizes. ╔═════════╦═════════╦═══════════╗ ║ memory used ║ structure size ║ memory increase ║ ╠═════════╬═════════╬═══════════╣ ║ 48 ║ 1 ║ ║ ║ 48 ║ 4 ║ 0 ║ ║ 48 ║ 8 ║ 0 ║ ║ 48 ║ 16 ║ 0 ║ ║ 48 ║ 24 ║ 0 ║ ║ 64 ║ 28 ║ 16║ ║ 64 ║ 32 ║ 0 ║ ║ 64 ║ 36 ║ 0 ║ ║ 64 ║ 40 ║ 0 ║ ║ 80 ║ 48 ║ 16 ║ ║ 96 ║ 64 ║ 16 ║ ║ 160 ║ 128 ║ 64 ║ ║ 288 ║ 256 ║ 128 ║ ║ 416 ║ 384 ║ 128 ║ ║ 544 ║ 512 ║ 128 ║ ║ 800 ║ 768 ║ 256 ║ ║ 1056 ║ 1024 ║ 256 ║ ╚═════════╩═════════╩══════════╝ Utilizing your info about the memory allocators and trying to come up with a more generic formula: const std::string memorySegmentName = "TrackOutput"; const std::size_t PADDING_MEMORY_FOR_SHARED_MEMORY_IN_BYTES = (2*sizeof(long))+(2*sizeof(long))+((memorySegmentName.length()+1)*sizeof(char))+(2*sizeof(long)); This seems to be the only way to account for the case where the structure is 1 byte in size and 48 bytes need to be allocated for it. Any suggestions? -- View this message in context: http://boost.2283326.n4.nabble.com/interprocess-How-to-know-how-much-memory-... Sent from the Boost - Dev mailing list archive at Nabble.com.
Dear Admin, It's ok if this does not reach the mailing list. I'm putting it up here as the answer, so that anyone browsing this Nabble forum will find it. The answer is here: http://stackoverflow.com/questions/33197889/boost-interprocess-mutex-crashes... -- View this message in context: http://boost.2283326.n4.nabble.com/interprocess-How-to-know-how-much-memory-... Sent from the Boost - Dev mailing list archive at Nabble.com.
On 10/08/2015 05:30 AM, nav999 wrote:
I have asked this question on stackoverflow <http://stackoverflow.com/questions/30642585/why-does-an-object-allocated-in-boost-interprocess-shared-memory-take-up-more-me> , but got no response. Hehe. This slipped through my [tag:boost] dragnet :)
I just answered there. Hope it helps. Seth
participants (3)
-
Ion Gaztañaga -
nav999 -
Seth