Re: [Boost-build] Help optimizing a slow BJAM build

From: Igor Nazarenko (igor.n.nazarenko_at_[hidden])
Date: 2008-04-04 13:52:18

> It's somewhat hard to say, since apparently the counters overflow.

Oh. How does one tell that they overflowed? In fact, is there some
documentation on the profiling output somewhere?

>> Looking at the source code of SORT in lists.c, it seems to me that the
>> function (list_sort) is more complicated than it needs to be, and may
>> be degrading to quadratic behavior on somewhat-pre-sorted lists.

> Why? It's standard merge sort, which is O(N*logN).

I'm not sure why: the code is a bit too much for me to follow in my
head. However, I hacked that function to print the number of elements
in the list and the number of split/merge iterations it does. I only
print it if the number of elements is above 1000. Here's the output:

$ bjam variant=debug,release
Fri Apr  4 10:26:42 2008
: entering list_sort for 46198 elements
Fri Apr  4 10:27:03 2008
: exiting list_sort after 5223 iters

Fri Apr  4 10:27:06 2008
: entering list_sort for 46198 elements
Fri Apr  4 10:27:28 2008
: exiting list_sort after 5224 iters

...patience...
...patience...
...patience...
...found 13910 targets...

So I'm pretty sure the SORT is screwed up somehow.

> install target only installs whatever you tell it to install, header
> dependencies are not processed.

Thanks a lot! I think I understand what happens. I have multiple
executables, and each of them depends on many shared libraries. That's
how the list of install targets grows so large.

> Can you try to commenting that call to sequence.unique, and see if that really
> improves anything?

I tried, but if I do that, BJAM never returns (I assume because it now
has to deal with full list of ~48000 targets, only a couple of which
are unique). Instead, I tried adding ' ECHO "foo" ;' before and after
every invocation of sequence.unique, and I can see that it takes a
while. See also hacked BJAM output above.

Thanks again!

Igor