Re: Sane regression testing

From: Beman Dawes (bdawes_at_[hidden])
Date: 2002-11-05 20:35:48

At 01:28 PM 11/5/2002, David Abrahams wrote:

>I've just checked in a modified version of tools/build/testing.jam on
>the "sane-testing" branch which I hope will have much more useful
>behavior than the previous version.

I've just tried several test cases, and all worked! Great job!

Only used the Filesystem Library path_test and the vc7 toolset due to time 
pressure. Note that path_test depends on the Filesystem Library build, 
which has three input sources.

Here were the test cases:

1) No existing bin directories. Both build and test ran fine. As expected, 
the residue files were a bit different: There was no path_test.success, and 
path_test.test contained the word "passed" instead of the path to the .cpp 
file. I've haven't looked yet to see if the lack of the path is a problem 
for reporting, but hopefully the information is trivially available 
somewhere else.

2) Missing source file for the library build. path_test.test was deleted, 
so the reporting programs would know that failure occurred. The stdout 
messages also indicated what had happened. Excellent!

3) Missing file restored. Worked fine; path_test.test was recreated, no 
other files were touched.

4) Introduced error into one of the library files, causing a library build 
compile to fail. path_test.test was deleted, so the reporting programs 
would know that failure occurred. Excellent! This is the case that has 
caused regression testing to report the wrong results in the past, so it 
was good news to see it working correctly now. We probably should figure 
out some other test cases just to be sure it is working for full coverage.

5) Fixed the error in the library file. The library build then worked, and 
path_test.exe was relinked, without first recompiling path_test.obj. Test 
was rerun. Exactly right behavior!

6) Ran bjam again, w/o changing any dependencies. As hoped, nothing was 
out of date, so no actions taken.

Because of non-Boost time pressures, it will be the end of next week before 
I can modify the status reporting programs to use "sane-testing".

Thanks for the improvements!

--Beman