Re: Bjam memory use -- Was: [BBv2] [msvc] Remove call to vcvars.bat

From: David Abrahams (dave_at_[hidden])
Date: 2005-10-06 10:29:16

"Reece Dunn" <msclrhd_at_[hidden]> writes:

> Rene Rivera wrote:
>
>>But AFAICT the only way to speed things up at this point is to reduce
>>the memory use. This is because of the way strings are managed in bjam,
>>and since *everything* is a string, or related to a string, there a
>>*lot* of strings floating around. A summary...
>
> Will this be helped when we move to Python, or will BBpy suffer similar 
> issues?

The reason bjam memory grows in that way is that _all_ strings are
"interned": they get allocated once and stored in a big hash table,
and the program tries very hard not to have duplicate allocations of
the same character sequence. They're also not refcounted, so they
live forever. And many of the programming idioms we've been forced to
adopt in Boost.Build, because of Jam's lack of a real type system,
involve conjuring up long strings that often -- in principle -- don't
need to persist very long. For example, if you want a list of lists in
Jam, you really need to concatenate all the elements in the inner list
with some delimiter character, since Jam can only represent
list-of-string directly.

Python doesn't have that sort of problem.

>>* We generate many slightly different strings because of how the class
>>system is implemented.
>
> I assume that native Python class support would help this issue, so is it 
> worth investing effort to improve this area now?
>
>>* Bjam naturally generated many strings because of the variable
>>expansion semantics, i.e. how variables are exploded into permutations.
>
> Sure. Is it possible to record those permutations and expand them when they 
> are evaluated? For example:
>
> local foo = "Hello " ;
> local bar = $(foo)"World" ;
>
> where bar will be bar[0] = &foo, bar[1] = "World".

I don't know what you mean. All strings are "recorded," as in
"interned."

>
> Are we going to place the Python engine inside bjam.exe 

According to posts I've read in the past 6 months, it's already done
(on a branch I guess?)

-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com