Re: [Boost-build] follow-up to my earlier post

Subject: Re: [Boost-build] follow-up to my earlier post
From: Rene Rivera (grafikrobot_at_[hidden])
Date: 2010-04-14 09:30:35

On 4/14/2010 2:30 AM, Ray Lambert wrote:
> On 04/13/2010 04:46 PM, Vladimir Prus wrote:
>> On Sunday 11 April 2010 21:36:37 Ray Lambert wrote:
>>> A few weeks ago I posted an inquiry regarding the detection of
>>> dependency changes during a build, which included a test case that
>>> demonstrates the issue. Here's my original post:
>>> <http://lists.boost.org/boost-build/2010/03/23253.php>
>> Ray,
>>
>> the caching of timestamp at a random moment during build process is
>> expected.
>> I don't know what effect such caching has on run-time performance,
>> compared
>> to calling 'stat' on files as necessary, and this is a code we've
>> inherited
>> from Perforce Jam. I don't have any high-level reason why this behaviour
>> should be like that, but probably don't have the time to experiment with
>> removing it right now either.
>>
>> Hope this at least clarifies the situation.
>
> Hi Vladimir,
>
> Thanks for your response.
>
> I'd like to clarify what I meant by caching time stamps.
>
> I don't know much about the BB internals but I envision at least two
> major phases: the parsing phase (when Jamfiles are read and tables are
> built) and the building phase.

That's correct.. Traditionally there are two phases, parse and make 
(there are actually more sub steps in there like dependency scanning 
though). Although with 3.1.18 Volodya added the ability to to makes at 
any time during the parsing.

> I've observed that the SHELL command executes before anything gets
> built. So it appears that it is executed either during the parsing phase
> or during some intermediate phase between parsing and building.

Yes, rules, which SHELL is, get executed during parse. Although there 
are additional callbacks for things like header scanning which do calls 
in the pre-make (for figuring out dependencies).

> This suggests that the SHELL command might be useful for running an
> external tool that might modify some dependencies before the actual
> build starts. This should work if time stamps are read during the
> building phase. However, this does not appear to be the case (as my test
> case demonstrates).

That's mostly correct, timestamps are read any time a file is looked up. 
This would include during the parse since one does greps, and target 
lookups. And the way bjam is written *all* file accesses are cached as 
otherwise it would be dog slow (it used to be dog slow in this regard 
long ago).

> My hope would be that BB could be made to delay the reading of time
> stamps until the latest moment possible, or at least until the build
> phase. This would allow the SHELL command (or something similar) to
> launch external tools before the build and to have their effects noticed
> by BB.

Not with some rather serious changes, and serious performance problems.

> As it is right now, the only way to run any tools like this prior to the
> build is to use a build script and invoke bjam from it. It would be much
> better (IMO) if one could invoke those sorts of things directly from BB.

There is one option you can use. If you set up the dependencies such 
that you create an intermediate "stamp"/"monitor" file between the 
targets you can manipulate the building such that you get what you want.

> Here are some examples from my experience:
>
> 1) I have a tool that retrieves the current source control revision
> number and updates a header file if it has changed so that it can be
> compiled into my app. The app needs to rebuild when the header file is
> updated, of course. I would also like to be able to rebuild this tool
> prior to running it, if its out of date.

One way I've done a similar task, in my case I need to generate a 
configuration header and get a coalesced set of headers, is to generate 
and copy to a separate headers location. But all that is done during the 
make stage and hence follows what I said above of having an intermediate 
file. It goes something like this:

* I have a bunch of source headers.
* With a set of configuration options specified during the parse phase.
* I generate a configuration header to the build location (since the 
configuration is build variant specific).
* I also copy the set of source headers to the build location of the 
config header.
* I build the code so that it uses the headers from the build location.

Dependency-wise it's:

   sources --> generated headers --> source headers

Hence the timestamps matter only for the creation of the generated 
headers. So the traditional make step works.

To translate this to your use case.. You could set things up so that 
instead of "updating" the header based on the revision you would instead 
create a new header in the build location and have the sources include 
it. And, yes, you can put in the intermediate tool rebuild because you 
make the generated header depend on the tool target.

> 2) I have a tool that reads a schema definition and generates a .cpp
> file and a .h file if they're out of date. The .cpp file needs to be
> compiled into a static lib and the .h file needs to trigger other files
> in the project to rebuild. As with (1), I would also like to be able to
> rebuild this tool prior to running it, if its out of date.

Same approach as above would work. To put it another way.. You should 
consider using the generated headers model of BBv2 instead of trying to 
do this during the parse phase.

You can see my use case at 
<https://svn.boost.org/svn/boost/sandbox/tools/build_extensions/ext/irrlicht.jam>. 
It uses a utility target that's defined here 
<https://svn.boost.org/svn/boost/sandbox/tools/build_extensions/ext/extension.jam> 
for generating the header files. But it just boils down to use the 
"make" BBv2 target and setting up the dependencies.

HTH

-- 
-- Grafik - Don't Assume Anything
-- Redshift Software, Inc. - http://redshift-software.com
-- rrivera/acm.org (msn) - grafik/redshift-software.com
-- 102708583/icq - grafikrobot/aim,yahoo,skype,efnet,gmail