Re: [Boost-build] Is there any way to prevent Boost.Build fromrecursively scanning header files for #include directives?

Subject: Re: [Boost-build] Is there any way to prevent Boost.Build fromrecursively scanning header files for #include directives?
From: Johan Nilsson (r.johan.nilsson_at_[hidden])
Date: 2009-04-27 07:06:39

Lin Luo wrote:
> Hi there,
>
> We would like to know that is there a way to limit the header files
> that Boost.Build recursively scans for #include directives to a
> particular directory or set of directories? I.e. I'd like it to
> recursively scan the header files within my project only. I know that
> the external dependencies are not going to change (and being Boost
> and Qt they're pretty big). I end up with around 50,000 targets in
> the dependency tree which takes a while to process (resulting in a
> 1-2 minute build time even if no files have actually changed).
>
> The only solution I've found so far is to take advantage of the
> INCLUDE environment variable (I'm using MSVC) - this means
> Boost.Build need not be informed of the include paths (I'm using the
> feature) and hence will not scan them. This seems a bit of a hack.
>
> I feel like I must be missing something obvious because I haven't
> been able to find other people experiencing similar problems, even
> though I ran into this almost immediately. The closest I've come is
> here.

I've mentioned something like this in the past. The general opinion seems to 
be that dependency scanning only takes an insignificant amount of time, 
which I really don't agree with (especially not for rebuilds after minor, 
local changes).

>
> Judging from the debug output (bjam -d 3) it also scans most of the
> header files more than once... I don't know if this means that they
> are added as dependencies more than once, but certainly the cost of
> loading a file and scanning the entire contents must add up?
>
> If I could tell it not to bother scanning a particular directory or
> set of directories in which I can guarantee the header files are not
> going to change, that would be perfect.

I've put together an "extended" C scanner that can optionally disable 
scanning of certain headers. Brief description:

- It determines which headers to skip by matching the "#include" statements 
using regex patterns. This means that you can't specify the actual directory 
on disk that should be skipped, which could cause problems.

- It terminates the header scanning when all found includes in a certain 
header matches the regex patterns. This obviously means that if a header 
that is skipped includes a header that should not be skipped, this won't 
work. Not a big problem in practice assuming that you've got your header 
dependencies in the correct direction.

- Due to the above restrictions I'd not recommend using this scanner during 
an actual release process.

Usage example (from Jamroot in one of my own projects):

--- Jamroot ---

...
local argv = [ modules.peek : ARGV ] ;
if --fastbuild in $(argv)
{
 echo WARNING: Fast (but unsafe) builds enabled ;
 #
 # Terminate scanning at boost headers and std library headers
 #
 import extcscan ;
 extcscan.set-scan-terminators ^boost/ ^[A-Za-z]+$ ;
}
...

--------------

I've included the extcscan.jam module to this post. To use it, add the 
containing directory to BOOST_BUILD_PATH.

Happy for any feedback.

Regarding speed improvements: For a specific project I went from 42s to 28s 
for a null build (no updates) from root when using "--fastbuild" (after 
warming up the disk cache before each attempt). The actual improvement 
obviously depends on how many skipped/static headers you actually use, so 
YMMV.

HTH,
Johan