Re: [boost] [interprocess] How to know how much memory is taken up by an object allocated in boost interprocess shared memory?

Subject: Re: [boost] [interprocess] How to know how much memory is taken up by an object allocated in boost interprocess shared memory?
From: nav999 (nkipe_at_[hidden])
Date: 2015-10-15 23:41:48

> On 08/10/2015 5:30, nav999 wrote:
> 
> > The output is:
> >
> > Before allocation = 649680
> > After allocation = 649632
> > Difference = 48
> > Size of SharedObject = 16
> > Size of SharedObject's temp instance = 16
> >
> > If the size of SharedObject and it's instance is 16 bytes, then how can
> the
> > difference in allocation be 48? Even if padding had automatically been
> done,
> > it's still too much to account for 3 times the size (for larger
> structures
> > it goes to 1.33 times the size).
> > Because of this, I'm unable to allocate and dynamically grow the shared
> > memory reliably. If SharedObject contains a list which grows
> dynamically,
> > that could add to the uncertainty of space allocation even more.
> >
> > How can these situations be safely handled? How can I know exactly how
> much
> > of memory to allocate?
> ... [show rest of quote]
> 
> Remember that any allocation, even with new or malloc, has a payload, in
> order to store information about how to deallocate that memory, how to
> merge that buffer with adjacent buffers, etc.. This happens with your
> system malloc (typically, 8-16 extra bytes per allocation plus extra
> alignment)
> 
> The memory allocator in shared memory has an overhead of 4-8 bytes (in
> 32 bits systems, 8-16 in 64 bit systems)
> 
> Then you need to store the number of objects in order to call
> destructors (you an allocate arrays) and you've done a named allocation.
> You need to store the string and metadata (a pointer to the string and
> maybe that this was a "named allocation" and not an anonymous or
> "instance allocation) in shared memory.
> 
> So 16 bytes of data + 8 bytes from the memory allocator + 8 bytes to
> store the pointer to the name plus metadata plus 12 bytes from the
> string "TrackOutput" (including null-end) plus rounding, you get 48 bytes.
> 
> The overhead is constant (except the name you supply) for each
> allocation. So the constant factor 1.33 is not for all objects only for
> small ones.
> 
> ¿How do you know how many heap you need to allocate elements in RAM?
> It's exactly the same problem (you allocate until malloc says there is
> no more memory).
> 
> You must also note that memory gets fragmented with allocations and
> reallocations, so even there is free memory, it maybe could not service
> your request because there is no contiguous memory big enough to fulfill
> your request. A shared memory can't be automatically expanded so
> fragmentation is a much bigger issue than with heap memory.
> 
> Ion 

Thank you Ion. I agree with pre-allocating memory, but even in such a case,
it helps to know how much of padding to allocate.
This is what I got when I tabulated values for multiple datastructure sizes.

╔═════════╦═════════╦═══════════╗
â•‘ memory used  â•‘ structure size â•‘ memory increase  â•‘
╠═════════╬═════════╬═══════════╣
â•‘                 48  â•‘                    1 â•‘                           â•‘
â•‘                  48 â•‘                    4 â•‘                        0 â•‘
â•‘                  48 â•‘                    8 â•‘                        0 â•‘
â•‘                  48 â•‘                  16 â•‘                        0 â•‘
â•‘                  48 â•‘                  24 â•‘                        0 â•‘
â•‘                  64 â•‘                  28 â•‘                       16â•‘
â•‘                  64 â•‘                  32 â•‘                        0 â•‘
â•‘                  64 â•‘                  36 â•‘                        0 â•‘
â•‘                  64 â•‘                  40 â•‘                        0 â•‘
â•‘                  80 â•‘                  48 â•‘                      16 â•‘
â•‘                  96 â•‘                  64 â•‘                      16 â•‘
â•‘                160 â•‘                128 â•‘                      64 â•‘
â•‘                288 â•‘                256 â•‘                    128 â•‘
â•‘                416 â•‘                384 â•‘                    128 â•‘
â•‘                544 â•‘                512 â•‘                    128 â•‘
â•‘                800 â•‘                768 â•‘                    256 â•‘
â•‘              1056 â•‘               1024 â•‘                   256 â•‘
╚═════════╩═════════╩══════════╝

Utilizing your info about the memory allocators and trying to come up with a
more generic formula:

const std::string memorySegmentName = "TrackOutput";
const std::size_t PADDING_MEMORY_FOR_SHARED_MEMORY_IN_BYTES =
(2*sizeof(long))+(2*sizeof(long))+((memorySegmentName.length()+1)*sizeof(char))+(2*sizeof(long));

This seems to be the only way to account for the case where the structure is
1 byte in size and 48 bytes need to be allocated for it.
Any suggestions? 

--
View this message in context: http://boost.2283326.n4.nabble.com/interprocess-How-to-know-how-much-memory-is-taken-up-by-an-object-allocated-in-boost-interprocess-sh-tp4680745p4680984.html
Sent from the Boost - Dev mailing list archive at Nabble.com.