Re: [Boost-build] BBv2 Slow Startup

Subject: Re: [Boost-build] BBv2 Slow Startup
From: Adder (adder.thief_at_[hidden])
Date: 2008-11-04 20:13:44

Hi,

First of all, thank you for your answer and your interest.

>> Here I am trying to compile Charles Petzold's "Analog Clock" 200-line
>> example program (which is shamelessly attached):
>> 
>>   - Running the compiler and the linker explicitly takes 328 ms.
>> 
>>   - Running bjam (with the attached Jamfile) takes 3969 ms,
>>     when there is no need to update anything ! That is: 4 seconds,
>>     without running the compiler and without running the linker.

VP> This is somewhat strange. Just to make sure we're on the same page, can you
VP> try building the examples/hello project, and measure time? For me, it's about
VP> 1 second for do-nothing build, with most of the time presumably spent for
VP> initial reading of all of Boost.Build source code and setting up.

I've attached the timings of building the "hello" project. I get long
start-up time here too. Just to make sure we're on the same page,
can you please let me know whether you're using Windows or Unix/Linux?

On one machine (PowerPC Mac with MacOS=Unix), I remember the building
starting up faster. On all Windows machines I've worked with, I suffer
the start-up problem. May I kindly ask if anyone else has experienced this ?

Disabling the following line in user-config.jam (or site-config.jam):
  use-project    /boost        : "$(BOOST_ROOT)" ;
causes the rebuild time of "hello" to change from 4400ms to 1600ms
(using the Jamroot file from SVN).

Disabling all other use-project directives (e.g. Loki & other libs)
causes the rebuild time to further decrease from 1600 ms to 1300 ms.

It seems to me that most of the time is spent in doing some kind of
processing of the Boost libraries.

I've uploaded the file-access log of building "hello" (in native
Process Monitor format and comma-separated-values format) here:
http://tinyurl.com/6xygvd .

VP> Are you using up-to-date versions of Boost.Build and Boost.Jam?

Yes, I have just svn update-d my "v2" folder and rebuilt bjam using
msvc-7.1.

>> On moderate projects, it takes 7 to 18 seconds to build 2 or 3
>> modified .cpp files, whereas building them from the IDE takes 1 to 2
>> seconds.
>> I've been able to reduce the build start-up time a little bit [...]

VP> You project depends on C++ Boost? We probably can arrange for the globbing
VP> not to happen unless Boost itself is installed. On C++ Boost in particular,
VP> a large portion of startup time is spend dealing with the filesystem.

Almost every project I work on depends on C++ Boost. Quite a few of them
require linking to some of the Boost libraries (not "filesystem";
mostly "thread" and "regex").

However, even if I make all-libraries null (and leave the globbing of
headers enabled), the gain is just 400 ms (down from 4200 ms). With
globbing of headers disabled, it's 400 ms (down from 2200 ms). For
projects larger than "Hello", it's about 400 ms down from at least
7-8 seconds.

I'm not sure what you mean by "dealing with the filesystem". I would
like to kindly ask you to provide a little bit of detail on the
time-consuming tasks that BBv2 must perform at start-up.

>> [...]
>> My questions are:
>> 
>> (1) Are there any plans or possibilities to make the system faster ?
>> What about caching the header dependencies (and only checking
>> the timestamps) ? 

VP> It's already done I believe.

Why then does bjam.exe open and completely read all the header files
(e.g. shared_ptr.hpp, detail/shared_ptr_nmt.hpp, etc. etc.) every time?

>> Or storing and loading precompiled Jamfiles (just as 
>> compilers store and load precompiled headers) ? Is the language of
>> choice (used to implement BBv2) to blame for the (lack of) speed ?

VP> Yes, we believe this is a major factor. In fact, Python port of Boost.Build
VP> was started to address this, and other issues.

I'm 100% sure you know best, but as a completely off-topic conversation,
may I kindly ask whether the C++ language itself has been considered for a
port of BBv2 ? :)

>> (2) Process Monitor shows a lot of attempts to open files with invalid
>> pathnames (such as E:\...\!-Home\AnaClock\<module@>\version.jam).
>> Can't these be avoided ?

VP> I'm not even sure where it's coming from :-(

Here's another interesting Process Monitor log line (among no less
than 67881 similar lines, containing "c-scanner" in a 20-hpp, 20-cpp
project that produces a total of 114822 file access lines):

"00:00:13.2301362","0.0000076","69633","bjam.exe","2788","QueryOpen","E:\Projects\MediumProjectWithNameMangled\<object(c-scanner)@747>boost\shared_ptr.hpp","NAME INVALID",""

(Bjam.exe is trying to open a file with an invalid name; the name is the
one after "QueryOpen".)

>> 
>> (3) The compilers I know (Borland, Visual, DMC, G++) allow the user to
>> specify multiple .cpp files in a single command: [...]

VP> This was discussed but nobody got to implement it.

I'd like to help, cause I think that compiling multiple .cpp files
from a single command line would make a significant difference, but I don't
think that I can, cause the system seems too complex for me ! Here I
am "investigating issues" and "profiling" with Process Monitor. :))
I wonder why nobody got to implement it...

Thank you again... Please do not read any disrespect between my lines.
There isn't any.

--
Yours truly,
Adder