Re: [Boost-build] Help optimizing a slow BJAM build

From: Vladimir Prus (ghost_at_[hidden])
Date: 2008-04-04 03:55:58

On Thursday 03 April 2008 22:34:25 Igor Nazarenko wrote:
> I have a relatively small project, currently consisting of ~200 C++
> files and headers. The project makes heavy use of BOOST 1.34, and also
> uses Xerces which adds another bunch of headers. The end result is
> that BJAM reports ~14,000 targets. Even when the build doesn't
> actually compile or link anything, BJAM is taking almost a minute on
> our fastest machine:
> 
> $ time bjam -j8 variant=debug,release
> ...patience...
> ...patience...
> ...patience...
> ...found 13910 targets...
> 
> real    0m56.029s
> user    0m55.802s
> sys     0m0.208s
> 
> I'm attaching the BJAM profile (obtained by running "bjam -n -d+10
> variant=debug,release > profile.txt").
> 
> >From looking at that profile, it seems that the major portion of time
> is spent in the following chain:
> class_at_install-target-class.collect-targets -> sequence.unique -> SORT.

It's somewhat hard to say, since apparently the counters overflow.

> Looking at the source code of SORT in lists.c, it seems to me that the
> function (list_sort) is more complicated than it needs to be, and may
> be degrading to quadratic behavior on somewhat-pre-sorted lists. 

Why? It's standard merge sort, which is O(N*logN).

> I 
> could try to change it to allocate an array of string pointers and
> sort it by standard qsort().
> 
> However, before I go there, two questions:
> - is it possible to tell Boost.Build system that the BOOST and XERCES
> headers are static, and there's no need to include them in the list of
> dependencies?

That's not going to help, assuming the slow part is what you've indicated.
install target only installs whatever you tell it to install, header
dependencies are not processed.

Can you try to commenting that call to sequence.unique, and see if that really
improves anything?

- Volodya