Re: [BBv2] fix for response file regeneration

From: David Abrahams (dave_at_[hidden])
Date: 2005-10-21 07:37:38

Reece Dunn <msclrhd_at_[hidden]> writes:

> David Abrahams wrote:
>> Reece Dunn <msclrhd_at_[hidden]> writes:
>> 
>>>I have the original Jam and Matt's Jam sources from Perforce
>>>(public.perforce.com Perforce connection) and am going to start
>>>looking at merging the changes into BJam.
>>>
>>>>I recall we had a long discussion about relative merits of those approaches, 
>>>>and did not decide anything. There are some usability questions, like 
>>>>removing or not removing files when build succeeds, or when it fails.
>>>
>>>We can get an implementation up and running, then modify it to best suit 
>>>our needs. Having a "native" solution would be a lot better than the 
>>>BBv2 implementation we currently have.
>> 
>> That just leads me to ask what's missing from the bjam core that we
>> can't implement the same behavior without resorting to extending bjam
>> specifically for response files?
>
> 1. The @(...) syntax is missing :). This would need to set up a special 
> "phantom" target, so the BBv2 logic can then generate the response file.

Yes, of course. The syntax counts, but that's now what I was
concerned about. If we move to a Scons engine we can process syntax
however we like but we'll want to create something like real targets
under the covers.

> 2. The main problem with the current response file code is that RSP 
> files are being treated as a target, so if you specify to always 
> generate the RSP file, it will recompile and relink the project that 
> uses them. Thus, we need some sort of "PHANTOM" or "NONTARGET" type that 
> will mean that we can have:
> NONTARGET $(rsp) ;
> ALWAYS $(rsp) ;
>
> where NONTARGET will not cause the specified target to be used in 
> determining whether other targets should be rebuilt. 

Isn't that what TEMPORARY is supposed to do? The idea is that you can
delete TEMPORARY object files after creating a library without causing
the library to be rebuilt.

> Thus:
>
> EXE --> CPP --> RSP
> \--> RSP
>
> will be seen as:
>
> EXE --> CPP
>
> (I like the name PHANTOM or PHANTOM_TARGET :))
>
>>>Q: What will happen when we port over to BBpy?
>> 
>> In the near term, we'll keep the same build engine, so nothing much
>> will happen.
>
> Ok.
>
> > In the long term, we might use Scons' build engine, but
>> that's just Python. We can extend it (if necessary) to do anything we
>> like.
>
> What impact would moving over to SCons have on people who are
> familiar with the BBv2 way of doing things?

It depends how much work we're willing to do to preserve the
"rule+actions" interface.

> What about the attempt to port BBv2 to Python? Would it be better to
> go down that route and have BBv2py?

No, the low-level Jam build engine is inscrutable at best. I don't
want to be saddled with maintaining it forever, even after translating
it to Python. Best to let the Scons team maintain the low-level build
engine.

>> However, I'd like to answer the question I posed above before
>> thinking of making that extension. Response files don't seem like
>> such a special case that they should require their own build engine
>> features.
>
> Ok. I have made an initial port of the code from Matt's response files, 
> but haven't had chance to test it beyond noting that it builds and 
> doesn't break existing BJam projects. I have noted that:
>
> 1. The BJam and Perforce Jam sources have now diverged to a point where 
> it is not trivial to integrate changes to BJam from PJam.

Old news. :)

> 2. Matt's Jam branch (MJam) has builtin support for the @(...) syntax.
>
> 3. MJam response files are created as temporary files so it won't be 
> easy to move them into bin/gcc/release style directories.

Why would we want to move them?

> 4. MJam uses tmpunix.c for creating NT/Unix temporary files: this would 
> require porting to the other platforms (e.g. MacOS).

?? MacOS is Unix

> 5. The code will require reformatting as the two sources use different 
> formatting conventions.
>
> So, as a result, I now agree with you: by adding phantom target support 
> (and optionally the @(...) syntax) we can fix the problem, but I have no 
> idea how to implement such functionality :(.

I'm not suggesting we shouldn't use Matt's code! I just want to make
sure we don't miss the opportunity to answer important questions by
adopting a quick fix.

> Doing this will mean that - with the existing BBv2 response file support 
> - response files are deleted if the target succeeds and are retained if 
> it fails. When rebuilding a failed target, the response file will be 
> regenerated. Thus this fixes the initial response file problem.

That sounds right. I think that's also what BBv1 does :)

> NOTE: On the changing properties - if we keep the response file, we can 
> use that as our basis for comparison as this stores includes and 
> defines. (I am not sure about support for changing other flags/settings).

I doubt that's the right way to go; it's very specific to toolsets
that can use response files. We want that capability everywhere.

-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com