Boost logo

Boost-Build :

From: Reece Dunn (msclrhd_at_[hidden])
Date: 2005-10-06 04:52:18

Rene Rivera wrote:
>Alexey Pakhunov wrote:
> > Reece Dunn wrote:
> >
> >>Is there a way that we can speed this up? Maybe preallocate an initial
> >>reserve space for strings? Or cache the memory as linked lists to save
> >>allocate+recopy time?
> >>When this is sorted, the load time should improve. Especially if you
>have a
> >>lot of configurations in user-config.jam and a large project base.
> >
> > Some profiling will definitely help. ;-)
>And on the new bjam the -d+10 option gives you that memory profiling info.


>But AFAICT the only way to speed things up at this point is to reduce
>the memory use. This is because of the way strings are managed in bjam,
>and since *everything* is a string, or related to a string, there a
>*lot* of strings floating around. A summary...

Will this be helped when we move to Python, or will BBpy suffer similar

>* We generate many slightly different strings because of how the class
>system is implemented.

I assume that native Python class support would help this issue, so is it
worth investing effort to improve this area now?

>* Bjam naturally generated many strings because of the variable
>expansion semantics, i.e. how variables are exploded into permutations.

Sure. Is it possible to record those permutations and expand them when they
are evaluated? For example:

local foo = "Hello " ;
local bar = $(foo)"World" ;

where bar will be bar[0] = &foo, bar[1] = "World".

>* Anything that processes string data, like header scanning in BBv2,
>will produce many strings.


>* Because the of the large number of strings the hash table caching them
> consumes much memory for the structure management.


>Here's a short dump, from running in Noel's test case, of the top memory
>And of course [TOTAL] is the sum of all memory use. As you can see it's
>a prodigeous 350Meg for a rather simple use case.

Yikes! I'll do a profile on my system when I get back to my computer which
will be next week.

>I don't have many ideas on how to improve the situation but here are two
>that might be promising:
>1. Create a native buffer functionality for which the memory can be
>manually managed.
>2. Reimplement the class system in some native form that doesn't require
> importing new rules for the instance methods.

Wont these be improved when we move to BBpy as Python has a native class
support and has things like sys.print and file management which should
improve things. It would be interesting to see how BBpy compares with BBv2
in terms of memory usage and speed.

Do you think memory consumption is one of the performance problems with

Are we going to place the Python engine inside bjam.exe (should this then be
bpython.exe or bpy.exe?) If so, will we have profiling (execution times and
memory) in this engine?

>And one last note, the above memory use is somewhat less than it was
>last week. As I made some minor changes to the class implementation code.

Great :)

- Reece


Boost-Build list run by bdawes at, david.abrahams at, gregod at, cpdaniel at, john at