Sane regression testing

From: David Abrahams (dave_at_[hidden])
Date: 2002-11-05 13:28:34

I've just checked in a modified version of tools/build/testing.jam on
the "sane-testing" branch which I hope will have much more useful
behavior than the previous version.

First, no attempt is made to avoid rebuilding tests which previously
failed but whose dependencies have not changed. This means that it
should be possible to find out which tests are failing without first
doing a "clean" or using "-a" on the command-line.

Output is improved. Here is an example of an expected-failure test:

gcc-C++-action bin\b.test\gcc\debug\runtime-link-dynamic\b.obj
b.cpp:6:2: #error this does not compile.
(failed-as-expected) bin\b.test\gcc\debug\runtime-link-dynamic\b.obj
**passed** bin\b.test\gcc\debug\runtime-link-dynamic\b.test

"run" tests now dump their output upon failure:

gcc-C++-action bin\fail-run-yes.test\gcc\debug\runtime-link-dynamic\c.obj
gcc-Link-action bin\fail-run-yes.test\gcc\debug\runtime-link-dynamic\fail-run-yes.exe
execute-test bin\fail-run-yes.test\gcc\debug\runtime-link-dynamic\fail-run-yes.run
====== BEGIN OUTPUT ====== 
returning result: 1
====== END OUTPUT ====== 

bin\fail-run-yes.test\gcc\debug\runtime-link-dynamic\fail-run-yes.exe > bin\fail-run-yes.test\gcc\debug\runtime-link-dynamic\fail-run-yes.output 2>&1 && copy bin\fail-run-yes.test\gcc\debug\runtime-link-dynamic\fail-run-yes.output bin\fail-run-yes.test\gcc\debug\runtime-link-dynamic\fail-run-yes.run || ( echo ====== BEGIN OUTPUT ====== && type bin\fail-run-yes.test\gcc\debug\runtime-link-dynamic\fail-run-yes.output && echo ====== END OUTPUT ====== && exit 1 )

...failed execute-test bin\fail-run-yes.test\gcc\debug\runtime-link-dynamic\fail-run-yes.run...

(Sorry, nothing I can do about the command output ;-))

If you want to see run-test output even for tests which succeed, pass
`--verbose-test' on the command-line:

gcc-C++-action bin\run-yes.test\gcc\debug\runtime-link-dynamic\c.obj
gcc-Link-action bin\run-yes.test\gcc\debug\runtime-link-dynamic\run-yes.exe
execute-test bin\run-yes.test\gcc\debug\runtime-link-dynamic\run-yes.run
1 file(s) copied.
====== BEGIN OUTPUT ====== 
returning result: 0
====== END OUTPUT ====== 
**passed** bin\run-yes.test\gcc\debug\runtime-link-dynamic\run-yes.test

Each test which succeeds leaves behind a .test file. If a test fails
or is skipped due to missing dependencies, the .test file will be
removed. Interrupting a test run in the middle may leave old .test
files lying around, but the next time you run the tests to completion,
the .test file will still be deleted unless the test succeeds.

"run" tests also leave behind a .output file containing a record of the
output. 

An expected-failure test will not succeed unless the specified stage
of construction fails. So, for example, if compilation fails during a
link-fail test, the test fails. The test will only succeed if linking
actually fails.

-- 
David Abrahams
dave_at_[hidden] * http://www.boost-consulting.com