Re: [Boost-build] BBv2 Slow Startup

Subject: Re: [Boost-build] BBv2 Slow Startup
From: Vladimir Prus (ghost_at_[hidden])
Date: 2008-11-04 03:44:32

On Tuesday 04 November 2008 10:31:00 Adder wrote:
> Hi there,
> 
> Boost Build v2 is a wonderful system, but unfortunately it takes a
> long time to start building stuff.
> 
> Here I am trying to compile Charles Petzold's "Analog Clock" 200-line
> example program (which is shamelessly attached):
> 
>   - Running the compiler and the linker explicitly takes 328 ms.
> 
>   - Running bjam (with the attached Jamfile) takes 3969 ms,
>     when there is no need to update anything ! That is: 4 seconds,
>     without running the compiler and without running the linker.

This is somewhat strange. Just to make sure we're on the same page, can you
try building the examples/hello project, and measure time? For me, it's about
1 second for do-nothing build, with most of the time presumably spent for
initial reading of all of Boost.Build source code and setting up.

Are you using up-to-date versions of Boost.Build and Boost.Jam?

> On moderate projects, it takes 7 to 18 seconds to build 2 or 3
> modified .cpp files, whereas building them from the IDE takes 1 to 2
> seconds.
> 
> I've been able to reduce the build start-up time a little bit (30
> seconds vs 60 seconds on cold-cache, 2 seconds vs 4 seconds on the
> "Analog Clock" example, etc.) by modifying the Boost Jamroot as
> follows:
> 
>   - "all-libraries" is manually assigned to:
> 
>       all-libraries =
>         date_time filesystem function_types graph iostreams math mpi
>         program_options python regex serialization signals
>         system test thread wave ;
> 
>   - "headers" is manually assigned to:
> 
>       local headers = boost/config.hpp ;

You project depends on C++ Boost? We probably can arrange for the globbing
not to happen unless Boost itself is installed. On C++ Boost in particular,
a large portion of startup time is spend dealing with the filesystem.

> I guess these modifications interfere with installing Boost headers;
> however, they avoid globbing from the Jamroot.
> 
> However, it still takes very long before the compiler is run... !
> 
> My questions are:
> 
> (1) Are there any plans or possibilities to make the system faster ?
> What about caching the header dependencies (and only checking
> the timestamps) ? 

It's already done I believe.

> Or storing and loading precompiled Jamfiles (just as 
> compilers store and load precompiled headers) ? Is the language of
> choice (used to implement BBv2) to blame for the (lack of) speed ?

Yes, we believe this is a major factor. In fact, Python port of Boost.Build
was started to address this, and other issues.

> (2) Process Monitor shows a lot of attempts to open files with invalid
> pathnames (such as E:\...\!-Home\AnaClock\<module@>\version.jam).
> Can't these be avoided ?

I'm not even sure where it's coming from :-(

> 
> (3) The compilers I know (Borland, Visual, DMC, G++) allow the user to
> specify multiple .cpp files in a single command:
> 
>   bcc32 -c Main.cpp Utilities.cpp Stuff.cpp
> 
>   is faster than
> 
>   bcc32 -c Main.cpp
>   bcc32 -c Utilities.cpp
>   bcc32 -c Stuff.cpp
> 
> Once the build start-up slowness is fixed, taking advantage of this
> feature of the compilers might speed up the actual compilation.

This was discussed but nobody got to implement it.

- Volodya