Bjam, BBv2 performance improvements.

From: Rene Rivera (grafik.list_at_[hidden])
Date: 2005-10-02 21:58:25

The variety of performance improvements I was working on are checked in. 
The basic improvement on the bjam side is caching of file query 
operations which is one place where Windows is horribly slow. There are 
other minor improvements on the BBv2 side. But we appear to be reaching 
a point of diminish returns. First some timings...

My desktop machine seems to be close enough to Noel's testing setup (I'm 
attaching the changes I made to it), the timing of BBv2-M10 on my 
machine are within 1 second of his times, that I feel safe giving these 
timings...

Linux, Ubuntu using GCC 3.4:

*** Full build ***

real 3m51.934s
user 2m57.176s
sys 0m53.522s

*** Incremental ***

real 0m21.986s
user 0m21.446s
sys 0m0.521s

real 0m22.093s
user 0m21.578s
sys 0m0.473s

real 0m22.088s
user 0m21.574s
sys 0m0.479s

real 0m22.015s
user 0m21.508s
sys 0m0.485s

real 0m22.106s
user 0m21.508s
sys 0m0.493s

*** Incremental lib ***

real 0m1.072s
user 0m0.989s
sys 0m0.082s

real 0m1.071s
user 0m0.991s
sys 0m0.080s

real 0m1.071s
user 0m0.996s
sys 0m0.073s

real 0m1.178s
user 0m0.991s
sys 0m0.083s

real 0m1.074s
user 0m1.000s
sys 0m0.072s

Windows, Win2K using VC-7.1 (the tests are from within the MSYS shell 
hence why the output is similar to Linux):

*** Full build ***

real 5m57.744s
user 0m0.010s
sys 0m0.020s

*** Incremental ***

real 0m20.690s
user 0m0.010s
sys 0m0.000s

real 0m20.489s
user 0m0.010s
sys 0m0.010s

real 0m20.460s
user 0m0.010s
sys 0m0.010s

real 0m20.459s
user 0m0.010s
sys 0m0.010s

real 0m20.460s
user 0m0.010s
sys 0m0.010s

*** Incremental lib ***

real 0m1.051s
user 0m0.010s
sys 0m0.010s

real 0m1.022s
user 0m0.010s
sys 0m0.010s

real 0m1.011s
user 0m0.020s
sys 0m0.000s

real 0m1.012s
user 0m0.010s
sys 0m0.000s

real 0m1.021s
user 0m0.010s
sys 0m0.010s

Basically it's about a 50% improvement overall. The Linux tests are a 
bit closer to his as there are less inherent performance problems with 
that OS.

One of the changes I did was to add more tracking to the "-d+10" option. 
In particular I added memory allocation tracking. This was to confirm a 
previous observation that the big bottleneck in performance is large 
memory use. Because of the way we manage strings it eats up a lot of 
memory and hence exercises one are of C run times and OS's that is 
usually weak. So if we want to significantly improve performance we need 
to reduce the our memory use. Any ideas on here are welcome :-)

One other change I did in some places to both reduce memory and to speed 
up execution time was in how some loops are written. There is one 
particular pattern that bjam seems to be slow at:

some-list = 1 2 3 4 ;
while $(some-list)
{
some-list = $(some-list[2-]) ;
}

And variations of that. Most likely the slowdown is in the loop test, 
but I'm not sure. The faster way to write the above is:

some-list = 1 2 3 4 ;
for local i in $(some-list)
{
}

This improvement applies even if doing n-wise iteration. That is 
pair-wise iteration is faster done as:

some-list = a 1 b 2 c 3 ;
local v = ;
for local i in $(some-list)
{
if $(v) { stuff ; v = ; }
else { v = $(i) ; }
}

-- 
-- Grafik - Don't Assume Anything
-- Redshift Software, Inc. - http://redshift-software.com
-- rrivera/acm.org - grafik/redshift-software.com
-- 102708583/icq - grafikrobot/aim - Grafik/jabber.org
 --------------040805080905020104000206 Content-Type: application/x-zip-compressed;
name="generate_libs.zip"
Content-Transfer-Encoding: base64
Content-Disposition: inline;
filename="generate_libs.zip"
[Attachment content not displayed.] --------------040805080905020104000206--