Re: [Boost-build] Is there any way to prevent Boost.Build fromrecursively scanning header files for #include directives?

Subject: Re: [Boost-build] Is there any way to prevent Boost.Build fromrecursively scanning header files for #include directives?
From: Lin Luo (chris.luolin_at_[hidden])
Date: 2009-04-27 07:25:18

Thanks Johan!

We'll analyze the Jamfile and give it a try.

Lin

-----Original Message-----
From: boost-build-bounces_at_[hidden]
[mailto:boost-build-bounces_at_[hidden]] On Behalf Of Johan Nilsson
Sent: Monday, April 27, 2009 11:07 PM
To: boost-build_at_[hidden]
Subject: Re: [Boost-build] Is there any way to prevent Boost.Build
fromrecursively scanning header files for #include directives?

Lin Luo wrote:
> Hi there,
>
> We would like to know that is there a way to limit the header files 
> that Boost.Build recursively scans for #include directives to a 
> particular directory or set of directories? I.e. I'd like it to 
> recursively scan the header files within my project only. I know that 
> the external dependencies are not going to change (and being Boost and 
> Qt they're pretty big). I end up with around 50,000 targets in the 
> dependency tree which takes a while to process (resulting in a
> 1-2 minute build time even if no files have actually changed).
>
> The only solution I've found so far is to take advantage of the 
> INCLUDE environment variable (I'm using MSVC) - this means Boost.Build 
> need not be informed of the include paths (I'm using the
> feature) and hence will not scan them. This seems a bit of a hack.
>
> I feel like I must be missing something obvious because I haven't been 
> able to find other people experiencing similar problems, even though I 
> ran into this almost immediately. The closest I've come is here.

I've mentioned something like this in the past. The general opinion seems to
be that dependency scanning only takes an insignificant amount of time,
which I really don't agree with (especially not for rebuilds after minor,
local changes).

>
> Judging from the debug output (bjam -d 3) it also scans most of the 
> header files more than once... I don't know if this means that they 
> are added as dependencies more than once, but certainly the cost of 
> loading a file and scanning the entire contents must add up?
>
> If I could tell it not to bother scanning a particular directory or 
> set of directories in which I can guarantee the header files are not 
> going to change, that would be perfect.

I've put together an "extended" C scanner that can optionally disable
scanning of certain headers. Brief description:

- It determines which headers to skip by matching the "#include" statements
using regex patterns. This means that you can't specify the actual directory
on disk that should be skipped, which could cause problems.

- It terminates the header scanning when all found includes in a certain
header matches the regex patterns. This obviously means that if a header
that is skipped includes a header that should not be skipped, this won't
work. Not a big problem in practice assuming that you've got your header
dependencies in the correct direction.

- Due to the above restrictions I'd not recommend using this scanner during
an actual release process.

Usage example (from Jamroot in one of my own projects):

--- Jamroot ---

...
local argv = [ modules.peek : ARGV ] ;
if --fastbuild in $(argv)
{
 echo WARNING: Fast (but unsafe) builds enabled ;  #  # Terminate scanning
at boost headers and std library headers  #  import extcscan ;
extcscan.set-scan-terminators ^boost/ ^[A-Za-z]+$ ; } ...

--------------

I've included the extcscan.jam module to this post. To use it, add the
containing directory to BOOST_BUILD_PATH.

Happy for any feedback.

Regarding speed improvements: For a specific project I went from 42s to 28s
for a null build (no updates) from root when using "--fastbuild" (after
warming up the disk cache before each attempt). The actual improvement
obviously depends on how many skipped/static headers you actually use, so
YMMV.

HTH,
Johan