Re: [jamboost] Regression test: link-fail problem?

From: Beman Dawes (bdawes_at_[hidden])
Date: 2002-07-09 11:09:25

At 09:28 PM 7/8/2002, David Abrahams wrote:

>This is exactly the sort of thing I was afraid would arise. AFAICT 
there's
>no simple way to put this in the build system without making core Jam
>changes.
>
>The basic problem looks like this. Given a dependency chain:
>
>a->b->c->d->e
>
>when building "e" fails, you want something to happen with the
>.test/.success/.failure files. These are associated with A. However, the
>build system just stops working on "a" the instant any of its 
dependencies
>fails to build.

Ah. That is a nice explanation, and gives me a better understanding of what 
is happening.

As long as a program scanning the output log and the residue left in the 
test directory can figure out where in the a->b->c->d->e chain a problem 
occurred, I may be able to deal with it.

> In Boost.Build v2, it will be much easier to deal with
>this, because we can traverse the dependency graph generated for A
>ourselves, and attach actions to erase the result files at all the 
leaves.

I've got some ideas about how to make the output from Boost.Build v2 
regression tests easier to used. I don't know it is premature, but I'll 
mention a few now:

* Having results spread over so many files (.run, .output, .success, 
.failure, .test, cout, and cerr ) is a major pain. If these could be 
consolidated into a single file, or at least a smaller number of files, 
that would make life much easier.

* It would be easier if the residue file name (or names, if they can't be 
combined) were always the same. For example, instead of it being 
abc.output for test abc, and then xyz.output for test xyz, call it 
test.output for all tests.

* Even if the cout/cerr output for a given test can't be combined with the 
other output and placed in the appropriate sub-directory, some format 
improvements would help a great deal. For example,

* Any output captured from a compiler, linker, test run, or whatever should 
be nested inside some easy to identify header and trailer. This is 
happening now for some C++ compile failures:

metrowerks-C++-action blah

compiler output here

...failed metrowerks-C++-action blah

But in other cases, the ...failed line is missing. Makes parsing very 
difficult, sometimes impossible.

* The "blah" in messages like the above is actually something like: 
..\libs\regex\build\bin\libboost_regex.lib\metrowerks\debug\runtime-link-dynamic\instances.obj...

While that may be needed for other purposes, for regression tests what is 
often needed is the name of the test which failed, and the name of the 
toolset involved. So a separate message may be needed.

* Top level needs identification on lib build failure. In the above 
example, some test needed the regex library, and that kicked off the regex 
build which then failed. What may be needed here is an explicit test type 
"lib-build" to go with "compile", "run", etc. Then lib build failures 
would show up on the status table in a very understandable way.

Comments?

--Beman