Re: [jamboost] Re: Sane regression testing

From: David Abrahams (dave_at_[hidden])
Date: 2002-11-05 20:48:46

Beman Dawes <bdawes_at_[hidden]> writes:

> At 01:28 PM 11/5/2002, David Abrahams wrote:
> 
> >I've just checked in a modified version of tools/build/testing.jam on
> >the "sane-testing" branch which I hope will have much more useful
> >behavior than the previous version.
> 
> I've just tried several test cases, and all worked! Great job!
> 
> Only used the Filesystem Library path_test and the vc7 toolset due to time 
> pressure. Note that path_test depends on the Filesystem Library build, 
> which has three input sources.
> 
> Here were the test cases:
> 
> 1) No existing bin directories. Both build and test ran fine. As expected, 
> the residue files were a bit different: There was no path_test.success, and 
> path_test.test contained the word "passed" instead of the path to the .cpp 
> file. I've haven't looked yet to see if the lack of the path is a problem 
> for reporting, but hopefully the information is trivially available 
> somewhere else.

If you want that info, I can put it in there. Please confirm that you
want it if so, and tell me if there's anything else you want to see in
the file.

> 2) Missing source file for the library build. path_test.test was deleted, 
> so the reporting programs would know that failure occurred. The stdout 
> messages also indicated what had happened. Excellent!
> 
> 3) Missing file restored. Worked fine; path_test.test was recreated, no 
> other files were touched.
> 
> 4) Introduced error into one of the library files, causing a library build 
> compile to fail. path_test.test was deleted, so the reporting programs 
> would know that failure occurred. Excellent! This is the case that has 
> caused regression testing to report the wrong results in the past, so it 
> was good news to see it working correctly now. We probably should figure 
> out some other test cases just to be sure it is working for full coverage.
> 
> 5) Fixed the error in the library file. The library build then worked, and 
> path_test.exe was relinked, without first recompiling path_test.obj. Test 
> was rerun. Exactly right behavior!
> 
> 6) Ran bjam again, w/o changing any dependencies. As hoped, nothing was 
> out of date, so no actions taken.

I can try to encapsulate this list of actions and expected results in
one of our Python testing scripts. Please see various scripts in
tools/build/test/*.py for examples. Specifically,
testing_primitives.py shows how I tested the basic components of the
new test system. I think these scripts are fairly self-explanatory,
and after I set up the cases above you should be able to add your own
without difficulty.

> Because of non-Boost time pressures, it will be the end of next week before 
> I can modify the status reporting programs to use "sane-testing".
> 
> Thanks for the improvements!

No prob. I'm really glad to have been able to clear this up.

-- 
David Abrahams
dave_at_[hidden] * http://www.boost-consulting.com