Re: [BBv2] fix for response file regeneration

From: Reece Dunn (msclrhd_at_[hidden])
Date: 2005-10-21 09:32:29

David Abrahams <dave <at> boost-consulting.com> writes:
> Reece Dunn <msclrhd <at> hotmail.com> writes:
> > David Abrahams wrote:
> >> Reece Dunn <msclrhd <at> hotmail.com> writes:
> >> That just leads me to ask what's missing from the bjam core that we
> >> can't implement the same behavior without resorting to extending bjam
> >> specifically for response files?
> >
> > 1. The <at> (...) syntax is missing :). This would need to set up a
special 
> > "phantom" target, so the BBv2 logic can then generate the response file.
> 
> Yes, of course. The syntax counts, but that's now what I was
> concerned about. If we move to a Scons engine we can process syntax
> however we like but we'll want to create something like real targets
> under the covers.

Sure.

> > 2. The main problem with the current response file code is that RSP 
> > files are being treated as a target, so if you specify to always 
> > generate the RSP file, it will recompile and relink the project that 
> > uses them. Thus, we need some sort of "PHANTOM" or "NONTARGET" type that 
> > will mean that we can have:
> > NONTARGET $(rsp) ;
> > ALWAYS $(rsp) ;
> >
> > where NONTARGET will not cause the specified target to be used in 
> > determining whether other targets should be rebuilt. 
> 
> Isn't that what TEMPORARY is supposed to do? The idea is that you can
> delete TEMPORARY object files after creating a library without causing
> the library to be rebuilt.

The problem is that if the target fails, the response file is kept around *but*
the action to regenerate the response file isn't called, so the old response
file is used. We could fix TEMPORARY so that the action will be called if the
target exists but in such a way that will prevent this from causing a rebuild of
the dependant target unless that target needs updating. I would be happy with
this solution.

> > > In the long term, we might use Scons' build engine, but
> >> that's just Python. We can extend it (if necessary) to do anything we
> >> like.
> >
> > What impact would moving over to SCons have on people who are
> > familiar with the BBv2 way of doing things?
> 
> It depends how much work we're willing to do to preserve the
> "rule+actions" interface.

I am not too bothered about the rule+action interface, more the things that BBv2
adds like the features, toolsets, etc.

> > What about the attempt to port BBv2 to Python? Would it be better to
> > go down that route and have BBv2py?
> 
> No, the low-level Jam build engine is inscrutable at best. I don't
> want to be saddled with maintaining it forever, even after translating
> it to Python. Best to let the Scons team maintain the low-level build
> engine.

So if I understand it:
* BJam will be BJam+Python allowing you to run Python code from
Jamfile/Sconsfile build scripts
* Scons will be responsible for the rule+action+target logic that Jam was
previously responsible for
* BBpy will add features, toolsets and all the fancy stuff I expect from BB :)

If I have the above right, then I am happy with that direction.

> > 1. The BJam and Perforce Jam sources have now diverged to a point where 
> > it is not trivial to integrate changes to BJam from PJam.
> 
> Old news. :)

:)

> > 2. Matt's Jam branch (MJam) has builtin support for the <at> (...) syntax.
> >
> > 3. MJam response files are created as temporary files so it won't be 
> > easy to move them into bin/gcc/release style directories.
> 
> Why would we want to move them?

Response files are created in bin/gcc/release/main.rsp with BBv2 whereas with
Matt's code (as far as I understand it) they will be in /tmp/file1214.rsp which
is not where BBv2 wants them (especially when we want to keep the response files
for failed targets).

> > 4. MJam uses tmpunix.c for creating NT/Unix temporary files: this would 
> > require porting to the other platforms (e.g. MacOS).
> 
> ?? MacOS is Unix

"Cannot say. Saying, I would know. Do not know, so cannot say." ;)

> > 5. The code will require reformatting as the two sources use different 
> > formatting conventions.
> >
> > So, as a result, I now agree with you: by adding phantom target support 
> > (and optionally the <at> (...) syntax) we can fix the problem, but I have
no 
> > idea how to implement such functionality :(.
> 
> I'm not suggesting we shouldn't use Matt's code! I just want to make
> sure we don't miss the opportunity to answer important questions by
> adopting a quick fix.

Agreed. I was refering to the PHANTOM targets/TEMPORARY fixes, not specifically
the @(...) implementation in Matt's code. However, it won't be trivial to do a
port given that we want to do something slighly different with the @(...) syntax
and the codebases have diverged so much.

> > Doing this will mean that - with the existing BBv2 response file support 
> > - response files are deleted if the target succeeds and are retained if 
> > it fails. When rebuilding a failed target, the response file will be 
> > regenerated. Thus this fixes the initial response file problem.
> 
> That sounds right. I think that's also what BBv1 does :)

If BBv1 has a fix for this situation (failed compile, change defines/includes,
RSP file not regenerated), we may need to look at porting the fix over to BBv2
but that depends on how different v1 and v2 are.

> > NOTE: On the changing properties - if we keep the response file, we can 
> > use that as our basis for comparison as this stores includes and 
> > defines. (I am not sure about support for changing other flags/settings).
> 
> I doubt that's the right way to go; it's very specific to toolsets
> that can use response files. We want that capability everywhere.

Understood. It was just an idea :).

- Reece