RE: [jamboost] Bjam memory use -- Was: [BBv2] [msvc] Remove call to vcvars.bat

From: Reece Dunn (msclrhd_at_[hidden])
Date: 2005-10-06 04:52:18

Rene Rivera wrote:
>Alexey Pakhunov wrote:
> > Reece Dunn wrote:
> >
> >>Is there a way that we can speed this up? Maybe preallocate an initial
> >>reserve space for strings? Or cache the memory as linked lists to save
> >>allocate+recopy time?
> >>When this is sorted, the load time should improve. Especially if you 
>have a
> >>lot of configurations in user-config.jam and a large project base.
> >
> > Some profiling will definitely help. ;-)
>
>And on the new bjam the -d+10 option gives you that memory profiling info.

:)

>But AFAICT the only way to speed things up at this point is to reduce
>the memory use. This is because of the way strings are managed in bjam,
>and since *everything* is a string, or related to a string, there a
>*lot* of strings floating around. A summary...

Will this be helped when we move to Python, or will BBpy suffer similar 
issues?

>* We generate many slightly different strings because of how the class
>system is implemented.

I assume that native Python class support would help this issue, so is it 
worth investing effort to improve this area now?

>* Bjam naturally generated many strings because of the variable
>expansion semantics, i.e. how variables are exploded into permutations.

Sure. Is it possible to record those permutations and expand them when they 
are evaluated? For example:

local foo = "Hello " ;
local bar = $(foo)"World" ;

where bar will be bar[0] = &foo, bar[1] = "World".

>* Anything that processes string data, like header scanning in BBv2,
>will produce many strings.

Sure.

>* Because the of the large number of strings the hash table caching them
> consumes much memory for the structure management.

Sure.

>Here's a short dump, from running in Noel's test case, of the top memory
>consumers:
>[snip]
>And of course [TOTAL] is the sum of all memory use. As you can see it's
>a prodigeous 350Meg for a rather simple use case.

Yikes! I'll do a profile on my system when I get back to my computer which 
will be next week.

>I don't have many ideas on how to improve the situation but here are two
>that might be promising:
>
>1. Create a native buffer functionality for which the memory can be
>manually managed.
>
>2. Reimplement the class system in some native form that doesn't require
> importing new rules for the instance methods.

Wont these be improved when we move to BBpy as Python has a native class 
support and has things like sys.print and file management which should 
improve things. It would be interesting to see how BBpy compares with BBv2 
in terms of memory usage and speed.

Do you think memory consumption is one of the performance problems with 
SCons?

Are we going to place the Python engine inside bjam.exe (should this then be 
bpython.exe or bpy.exe?) If so, will we have profiling (execution times and 
memory) in this engine?

>And one last note, the above memory use is somewhat less than it was
>last week. As I made some minor changes to the class implementation code.

Great :)

- Reece