Re: [boost] Multithreaded Scaling Issues with regex

From: Matt Warner (matt_at_[hidden])
Date: 2003-03-28 10:36:39

>  > I am writing a multithreaded Apache log parser that uses the Boost
>>  1_29_0 regex split function to separate elements in the entry. Each
>>  thread parses a separate log file. The code seems to be working
>>  correctly on a 1-CPU system, but when I use a 14-CPU Sun server, I
>>  see massive locking (LCK column of prstat -amLvu username), and
>>  performance suffers horribly (as measured by the lines processed per
>>  second). I spent a lot of time checking to see where the locking was
>>  occurring. I went so far as to compile the code with Sun's Forte 6u2
>>  and use their analysis tools to identify the problem area. I've
>>  compiled all code (including Boost) with both gcc 3.2.2 and Forte to
>>  create 64-bit binaries, if that makes any difference.
>>
>>  If I read the Forte analysis tools correctly, the place I'm seeing
>>  all the locking is the call to malloc in the void *operator
>>  new(unsigned long), which is called by
>>  boost::re_detail::match_results_base and _priv_match_data. Those are
>>  in turn called by query_match_aux, which is called by reg_grep2.
>>  Assuming I'm reading it right...
>>
>>  At this point it seems like the issue is either with the library or
>>  my usage of it. Has anyone seen this before? Any pointers on what I
>>  may be doing wrong and how to fix it would be appreciated.
>
>The looking is occurring in your runtime library rather than boost.regex as
>such.  You have two choices:
>
>1) Use a custom allocator for the match_results class instance that you are
>using that uses thread-specific memory pools.
>
>2) Wait for the next release (probably still a couple of months away), which
>will use much less dynamic memory allocation (almost none at all in
>recursive mode).
>
>John.
>
Thanks for the quick response. Any chance that the revised memory 
allocation is in a current beta version?

Matt