Re: [boost] Scons users?

From: David Abrahams (dave_at_[hidden])
Date: 2005-09-21 11:09:20

Vladimir Prus <ghost_at_[hidden]> writes:

> Hi Steven,
>
>> In a weird way, I'm comforted (*not* in a competitive, ha-ha way)
>> to find out that Boost.Build has some performance issues. SCons has
>> gotten beaten up pretty badly for performance over the last year or so,
>> and people have been pretty quick to point fingers at causes based on
>> a lot of assumptions and little data: "It's because it's in Python."
>> "It's because it's MD5 checksumming the contents." "It's because it's
>> scanning file contents for dependencies." Etc., etc., etc.
>>
>> Well, I've put in a *huge* amount of time over the last year profiling and
>> trying to optimize SCons, and although I've found a number of things to
>> clean up, there simply aren't any smoking-gun bottlenecks. Seeing data
>> like this strongly suggests that the performance issues of SCons and
>> Boost.Build are largely because they're trying to solve a much more
>> complicated set of problems than people normally expect from a build tool:
>> correct cross-platform builds with multiple variants and tool chains.
>
> I'm not sure I agree. For Boost.Build, lots of performance is spend into 
> innocent functions like, ehm.. path.join, which suggests that the problem is 
> that bjam spends too much time interpreting the code. 

1. Python's interpreter is a *lot* more sophisticated thank bjam's,
and more highly optimized.

2. Perhaps this statistic suggests that we're overusing path syntax to
represent things that really ought to be proper collections?

>> But you're exactly right, there are lots more developers out there
>> than there are buildmasters, so this perception of performance will
>> undoubtedly dominate for some time. In the meantime, I've been trying to
>> back SCons away from *insisting* on a full DAG of the whole source tree
>> every time, and find ways to allow the user to partition the DAG safely.
>> There's probably a comfortable middle ground to be found somewhere.
>
> For the record, lazy nature of Boost.Build is especially handy there. In 
> Boost.Build V1, going to "status" directory of Boost tree (with lots of 
> targets) and running "bjam -n just_one_target" takes roughly the same time as 
> just "bjam -n". For V2, the dependency graph is built only for requested 
> target which gives serious speedup in that case.

Yeah, that's nice. Its potential impact is somewhat limited if you
have only one target you ever really care about, but that target has
lots of build dependencies. That's a pretty common development
scenario.

One thing I'd like to see is a breadth-first dependency checker. That
would allow us to discover early on that something needs to be rebuilt
in most of the common cases, so we could start rebuilding it
immediately and continue the dependency check while that build is
running. That would be a win even on most uniprocessor systems, and
on SMP or cluster builds it would kick *ss. Of course, the structure
of Jam is incompatible with that approach. I believe that Scons is
more adaptable to it.

-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com