Bjam memory use -- Was: [BBv2] [msvc] Remove call to vcvars.bat

From: Rene Rivera (grafik.list_at_[hidden])
Date: 2005-10-04 17:29:19

Alexey Pakhunov wrote:
> Reece Dunn wrote:
> 
>>Is there a way that we can speed this up? Maybe preallocate an initial 
>>reserve space for strings? Or cache the memory as linked lists to save 
>>allocate+recopy time?
>>When this is sorted, the load time should improve. Especially if you have a 
>>lot of configurations in user-config.jam and a large project base.
> 
> Some profiling will definitely help. ;-)

And on the new bjam the -d+10 option gives you that memory profiling info.

But AFAICT the only way to speed things up at this point is to reduce 
the memory use. This is because of the way strings are managed in bjam, 
and since *everything* is a string, or related to a string, there a 
*lot* of strings floating around. A summary...

Bjam caches all strings, with some internal exceptions, into a hash 
table. This is so it can reuse repeated strings to reduce memory use 
(can't even imagine what the mem use would be without this).

* We generate many slightly different strings because of how the class 
system is implemented.

* Bjam naturally generated many strings because of the variable 
expansion semantics, i.e. how variables are exploded into permutations.

* Anything that processes string data, like header scanning in BBv2, 
will produce many strings.

* Because the of the large number of strings the hash table caching them 
consumes much memory for the structure management.

Here's a short dump, from running in Noel's test case, of the top memory 
consumers:

20100 10264 420 0.020896 4892626 243 
generators.try-one-generator
5050 7992 210 0.041584 5097820 1009 
class_at_action.actualize
20385 210 120 0.005887 6205020 304 
class.xinit
20100 260 190 0.009453 7073221 351 
generators.viable-source-types-for-generator
15100 7992 240 0.015894 7270594 481 
class_at_virtual-target.actualize
15100 7992 110 0.007285 8864430 587 
class_at_abstract-file-target.actualize-no-scanner
5100 950 460 0.090196 13794634 2704 
generators.find-viable-generators
5050 1922 511 0.101188 17555491 3476 
common.response-file
5837 24524 340 0.058249 17563141 3008 
modules.import
10100 910 540 0.053465 20413007 2021 
print.text
5050 1662 880 0.174257 30214826 5983 
toolset.set-target-variables
5000 702 390 0.078000 32308920 6461 
scanner.propagate
5000 2373 611 0.122200 46334519 9266 
class_at_c-scanner.process
56358664 11466 11466 0.000203 64638420 1 [OTHER]
0 30352 0 0.000000 384811960 384811960 [TOTAL]

(Sorry if that wraps)

In the above [OTHER] is the memory the hash table itself is using. Mind 
you this is *not* the working set, but the total allocations. Some 
percentage of the above get's deallocated shortly after it get's 
allocated, but it's a *small* percentage.

And of course [TOTAL] is the sum of all memory use. As you can see it's 
a prodigeous 350Meg for a rather simple use case.

I don't have many ideas on how to improve the situation but here are two 
that might be promising:

1. Create a native buffer functionality for which the memory can be 
manually managed. This would allow some strings to move out of the cache 
and be allowed to be freed soon after they are no longer needed. The 
"print" functionality would make use of this. As so would any CAT like 
builtin. It might be possible to make header scanning also use this. 
This is not a real savings in memory use, just on lowering the impact on 
other string use.

2. Reimplement the class system in some native form that doesn't require 
importing new rules for the instance methods. This would be a _big_ 
win as it would remove by some factors the number of rule names floating 
about in the string table. I haven't had time to think exactly how to 
implement this change so not sure if it would even work, or I'm even 
close in thinking hwo the class system is implemented :-)

And one last note, the above memory use is somewhat less than it was 
last week. As I made some minor changes to the class implementation code.

-- 
-- Grafik - Don't Assume Anything
-- Redshift Software, Inc. - http://redshift-software.com
-- rrivera/acm.org - grafik/redshift-software.com
-- 102708583/icq - grafikrobot/aim - Grafik/jabber.org