Re: [Boost-testing] Re: [regression testing] BBv2?

From: Vladimir Prus (ghost_at_[hidden])
Date: 2005-04-14 02:57:26

Caleb Epstein wrote:

> 
> On 4/13/05, Vladimir Prus <ghost_at_[hidden]> wrote:
>> Right at the moment I'm tracking down the remaining differences between
>> V1 and V2 test results on Linux. Once that done -- hopefully in a day or
>> two -- it would be really great if you could run tests with V2 to verify
>> that my system is not the only one where V2 works OK.
> 
> I'd be happy to run tests w/BBv2 also, especially if the
> results-collection will work properly with parallel builds (e.g. -jX).
>  Currently it takes >12 hours to do a single regression testing run on
> a 24-way SPARC machine I have access to.  If parallel builds worked, I
> could get these done in a considerably shorter time.
> 
> I suspect this hasn't been tackled yet though.

Right, it was not. I was thinking about this, though, and now believe this
should be addressed outside of jam. Specifically, we need a utility to run
a command, that can 
- store the command output to a specified file (avoiding the problem with
interleaved -jX output)
- record elapsed time and memory consumption and kill the program when some
  limit is reached. Or maybe attach to the program via debugger and produce
  backtrace
- maybe directly generate XML file

jam has a variable JAMSHELL. Say, if you set it to "my_runner %", then
bjam will execute all commands by running the "my_runner" command with the
text of the command to be executed. We can extend this to allow passing the
action name, so JAMSHELL would be set to "my_runner --action="$" %", and
bjam will be invoking my_runner like this:

  my_runner --action="gcc.compile.c++ bin/whatever.o" 
      "g++ -o bin/whatever.o whatever.cpp"

After running the command and capturing the output, we'll get the exactly
same results as process_jam_log uses, but it will work with -jX, and
'my_runner' can do many other things.

There could be issues on windows due to limited command line length. But
hopefully, boost test don't produce such long command lines.

Comments?

- Volodya