Re: [BBv2] [bjam] Native response file support

From: Reece Dunn (msclrhd_at_[hidden])
Date: 2005-10-31 03:20:47

Rene Rivera wrote:
> Reece Dunn wrote:
> 
>>The results of the discussion on the "[BBv2] fix for response file 
>>regeneration" thread was to have builtin support for response files 
>>using the @(...) syntax based on Matt Armstrong's perforce Jam branch.
>>
>>The bjam-response.diff patch adds @(...) syntax support for NT builds of 
>>BJam. This is based on Matt's code but was written from scratch and is 
>>improved from my initial submission. When you use @(...) the code (in 
>>variable.c) expands the variables in the '...' to a temporary file and 
>>then outputs the location of that file.
> 
> How does Matt's version of the syntax @() work? I'm asking because I was 
> going to apply, with some mods, your patch but I decided not to because 
> I have some concerns. Instead I just refactored the tmpdir function in 
> preparation, with my own simpler implementation (both Windows and Unix).

Matt's code created a cmd_string method that was a duplicate of 
var_string, but with a check for @(). It reads the string between the 
parentheses, expands that to a temporary buffer using var_string and 
then writes that to a temporary file. The temporary files are (in Matt's 
code) kept in a linked list and managed by a tmpunix.c file.

I simplified Matt's code somewhat, moving the response file logic into 
var_string. The var_string_file function was added to avoid writing the 
strings to a buffer first.

> My basic concern is that as you implemented @() is the limited response 
> file specific functionality. I really loath adding generic looking 
> syntax that isn't generic. The limitation is that there is no control 
> over what the name of the temporary files are. In particular they are 
> fixed to have an ".RSP" extension.

This was based on what Matt's code did, but having the ability to name 
the file is a good idea.

> Since I'm not one to usually be 
> critical without having some suggestions one how to fix things here's 
> what I'm thinking...
> 
> The "@()" is ultimately a "here document" functionality. With that in 
> mind I'd rather have it be as flexible as possible. My suggestion is to 
> use the standard variable expansion syntax to manage control of where 
> and how the document is put together. Like so:
> 
> 1) For a temporary file:
> 
> @($(TMPFILE):S=.txt:E="This is some content.")
> 
> And like your changes the @() would evaluate to the name of the 
> temporary file.

I suggest that we also have the ability to control the lifetime of the 
file. Note that your syntax will complicate parsing and processing:

@($(<[1]:W):F=.rsp:E=""$(>)" "-I$(INCLUDES)" -D$(DEFINES)")

vs.

@("$(>)" "-I$(INCLUDES)" -D$(DEFINES))

How about:

@("$(>)" "-I$(INCLUDES)" -D$(DEFINES))="$(<[1]:W).rsp"

> 2) For doing std output:
> 
> @($(STDOUT):E="Some console output")
> 
> This would allow writing actions that take standard input without having 
> to resort to echo. For example:
> 
> actions xyz
> {
> @($(STDOUT):E="$(>)") | sort > $(<)
> }
> 
> 3) Similarly for STDERR.

How about:

@("$(>)" "-I$(INCLUDES)" -D$(DEFINES))=$(STDOUT)
@("$(>)" "-I$(INCLUDES)" -D$(DEFINES))=$(STDERR)

> Note: the $(TMPFILE) and $(STDOUT) could just as well be TMPFILE and 
> STDOUT (without the $()). But the $() seems more fitting to me.

Sure. I like the $() syntax as well.

> The above has enough flexibility to be useful in many context. In 
> particular, in addition to the response file use, it would allow for a 
> less kludgy implementation of the "print" module. It would no longer 
> need to do the funky string escape manipulations to get over the shell 
> echo command use. And because of that reduce the memory consumption 
> (print.text is one of the top 3 memory consuming rules in BBv2). And 
> apply that to any other place where BBv2 needs to escape strings so they 
> can be handed to the shell echo or some other command that can handle 
> std input.

That would be really neat. Note that in my code (with var_string_file), 
each string is separated by a newline character, not a space. Otherwise, 
you get a long like that is not too readable in the response file. If 
this is an issue, we sould pass the separator in the @(...) syntax, e.g.:

@("$(>)" "-I$(INCLUDES)" -D$(DEFINES)|$(nl))="$(<[1]:W).rsp"
^^^^^^

- Reece