Re: [jamboost] Performance degradation vs number of files

From: Vladimir Prus (ghost_at_[hidden])
Date: 2003-10-13 10:01:41

Pedro Ferreira wrote:
> Hi all,
>
> I'm using BBv2 on Windows XP.
> I have a project with a couple of hundreds of cpp files and bjam is taking
> an enormous amount of time just to start building it. To try and figure out
> why, I've set up a simple evaluation script which generates empty cpp files
> and adds them to a Jamfile with a lib target.
>
> Running "bjam -d -d+10" renders this:
>
> # files 1 Elapsed: 0.451 seconds
> # files 5 Elapsed: 0.621 seconds
> # files 10 Elapsed: 0.811 seconds
> # files 50 Elapsed: 2.684 seconds
> # files 100 Elapsed: 5.718 seconds
> # files 200 Elapsed: 11.517 seconds
> # files 300 Elapsed: 17.435 seconds
> # files 500 Elapsed: 47.418 seconds
> # files 1000 Elapsed: 154.132 seconds

Interesting data!

> Does anybody have an idea why the behaviour seems to become exponential
> after 300 targets? Including 100 header files in each cpp file increased
> the time by 10-20%.

Not yet...

> I found it odd that the highest net time is caused by numbers.increment:
> files gross net # entries name
>
> 1: 0 0 217 numbers.increment
> 500: 17334 17004 19182 numbers.increment
> 1000: 70299 69958 38182 numbers.increment
>
> Any pointers on how to optimize this would be appreciated.

Hmm... that's interesting. While the #entries seems to grow by factor of 2 
between 500 and 1000 targets, time grows much higher. Possibly, 
numbers.increment just does not scale. 

In fact, all of the profiles I've recieved after asking for profile data 
recently has numbers.increment high in the list of top consumers. I think it 
would be best to reimplement in using builtin (i.e. in C). Probably you might 
try it? If you're out of time for now, no problem --- this is the first thing 
I'm about to optimize, anyway, just not sure about exact date.

- Volodya