Re: [boost] Regex allocator support

Subject: Re: [boost] Regex allocator support
From: Fred Sundvik (fred.sundvik_at_[hidden])
Date: 2009-12-03 05:39:11

>> We recently had to use the regular expression library in conditions
>> where there was no available physical memory, for reporting resource
>> consumption when our console game runs out of memory. This wasn't
>> possible however due to the fact that the basic_regex does some
dynamic
>> allocations and doesn't provide any allocator specialization,
allthough
>> match_results for example does. So we modified the boost code a
little
>> bit and added a new template paramter for the allocator, and made
>> constructors that takes allocators as parameters.
>>
> <snip>
>>
>> So my question is, what are the chances that theese kinds of
>> modificiations gets implemented in the actual library, and possibly
in
>> the next c++ standard library as well? I consider it a quite big
design
>> flaw to not have any control of the internal allocations, so for me
this
>> a quite big thing.
>
> +1; I agree with the OP's sentiments, though I wouldn't say it's
> necessarily a "big" design flaw.  However, I've never used the
> Boost.Regex library, and I don't know if the next standard specifies
> allocator parameters.  I gather it doesn't?

Sigh... no.

The original regex++ library upon which the Boost version is derived
*did* 
have allocators for everything.  But during review folks felt quite
strongly 
that:

a) It was overkill.
b) Regex should be free to manage it's own memory - it's not a container
- 
it's needs are much more complex than that - so it should be free to 
optimize memory allocation and caching as it sees fit.

As a result this facility was removed.  Rightly or wrongly, no one on
the 
stds committee questioned that design decision.

Some specific comments on the other issues:

>The mem_block_cache system was a bit more complicated. Initially we 
>disabled
>it completely, but that obviously wasn't the best solution. So we
figured
>out, that we can use a custom allocator as default, instead of
>std::allocator. That custom allocator then had to be specialized for
the
>mem_block_nodes, and the funcion calls to get_mem_block replaced by
>allocator calls. Our implementation isn't ideal, though as we used
>mem_block_node as the specialization, and to pass the correct size to
the
>allocator, we allocate  BOOST_REGEX_BLOCKSIZE / sizeof(mem_block_node)
of
>them, this way it works for allocators that don't specialize for
>mem_block_node(but only if the size is dividable by
>sizeof(mem_block_node) ). Better would probably be to make a new type
that
>is BOOST_REGEX_BLOCKSIZE big.

Can you not just define BOOST_REGEX_RECURSIVE and use the stack based 
implementation that does away with that altogether?

>We also found another problem with allocation, the match_result::format
>functions only takes string_type, that isn't specialized by any
allocator 
>as
>they input. This made it impossible to call them with strings using
>different allocators. So we added specializations of those functions
for
>different allocators and traits. There are probably more functions like
>that, but since our code doesn't use them there was no need to change
>anything.

The current Trunk allows any string, container or character type, plus 
function objects to be used as arguments to regex_format/regex_replace
etc, 
which should avoid that problem.

>We recently had to use the regular expression library in conditions
where
>there was no available physical memory, for reporting resource
consumption
>when our console game runs out of memory.

I'm not sure I understand the issue here, that operator new doesn't
report 
out of memory conditions?  What does your allocator do that's different
to 
operator new, and why not replace global new and delete with calls to
your 
custom allocator?  Just trying to get a handle on the issue...

>So my question is, what are the chances that these kinds of
modifications
>gets implemented in the actual library, and possibly in the next c++
>standard library as well?

As noted above, I was asked to remove this feature during review, so I'm
not 
*that* keen to put it back in!  As far as the standard is concerned it's

basically a done deal at this stage in the process - too late for such a
big 
change.  WRT Boost I'd really like to get a handle on the issue better 
before making judgment - you seem to be one of very few people actually 
using custom allocators ;-)

Regards, John.

_______________________________________________
Unsubscribe & other changes:
http://lists.boost.org/mailman/listinfo.cgi/boost