[Boost-users] [pool] pool allocator: customizing and memory usage

From: Lang Stefan (SLang_at_[hidden])
Date: 2008-02-19 08:25:33

Hello all,

I've been trying to integrate pool allocators for lists within our
project, but early test runs turned up quite a surprising number of
questions and problems. I've tried to resolve those issues by closely
regarding the documentation, debugging, and - finally - checking this
list for threads that potentially answer my questions, but unfortunately
I found that on this list more questions were raised than answered.

Specifically, I noticed that Stephen Cleary doesn't seem to  frequent
this list, or has stopped doing so some 5 or 6 years ago.

Nevertheless maybe there are other people out there who can answer one
or more of my questions, so here I go:

1. When using fast_pool_allocator for std::list<> I see no way to
influence the blocksize being used for the next allocation. Since our
application handles huge lists I cannot have fast_pool_allocator always
simply double the next block's size ad infinitum! The pool class does
offer methods to inquire and set the next_size property, but
unfortunately, the singleton_pool underlying fast_pool_allocator is not
accessible through the allocator interface!
Is there a recommended way to set or limit next_size for pool
allocators?

I have found a workaround to achieve this (see below) but am not very
comfortable doing it this way.

2. I am under the impression fast_pool_allocator heavily leaks memory!
As far as I can tell, it works properly in the beginning, but uses up to
7 times the memory a std::list<T,std::allocator<T> of the same size
uses! I am talking millions of list nodes and gigabytes of required
memory here, when it should be only some 200 MB of used memory. I've ran
a test to find out how much memory exactly was being allocated in
comparison to the amount being requested by inserting into the list and
found the factor to be exactly 7.0 .

Can anyone confirm this observation? Is there a fix, or anything I can
do to prevent it?

3. Also the release_memory method doesn't appear to work reliably,
locking up huge amounts of memory even after the list using the pool ran
out of scope!

How can I force a singleton_pool to reliably release memory - short of
stopping the application!?

4. The purge_memory method occasionally throws runtime errors, and when
it does not it also doesn't appear to release any memory.
It is possible this is due to inaccurate usage, but I really haven't
delved into this, because release_memory() is what I actually need.

Any specific things I need to do, any conditions I need to check, before
a call to purge_memory?

Workarounds:

I defined my own allocators and then specialized the
pool<>::malloc_need_resize() method to get more control and info on what
was happening internally. So far all the problems I noticed (and more)
have been confirmed. I am also quite sure I used all the pool and pool
allocator interfaces properly (except maybe purge_memory, as noted
above), and yes, I am aware that pool<> does allocate blocks in
multiples of a size that is big enough to contain several instances of
the object being managed (list nodes in this case).

I have considered fixing one or more of the issues above by similarily
defining my own pool<> method specializations, but I am very
uncomfortable doing so since I don't fully understand the inner workings
of the pool classes, and am not very experienced in metaprogramming.

At this point I am about to give up. It is a shame however - I notice an
increase of allocation/deallocation speed by a factor of 6-8, although
in hindsight this might also be in part due to the fact release_memory
simply doesn't work... 

Is there anyone out there using pool_allocator or fast_pool_allocator,
and willing to share advice?

Cheers,
Stefan