Re: [jamboost] Re: Sane regression testing

From: David Abrahams (dave_at_[hidden])
Date: 2002-11-20 10:45:54

OK, Beman, there are regression tests for this in the CVS. See
tools/build/test/v1_testing.py. I'd really like to move sane-testing
to the main trunk soon. Do you think you can update the XML generation
stuff in the near future?

Beman Dawes <bdawes_at_[hidden]> writes:

> At 01:28 PM 11/5/2002, David Abrahams wrote:
>
> >I've just checked in a modified version of tools/build/testing.jam on
> >the "sane-testing" branch which I hope will have much more useful
> >behavior than the previous version.
>
> I've just tried several test cases, and all worked! Great job!
>
> Only used the Filesystem Library path_test and the vc7 toolset due to time 
> pressure. Note that path_test depends on the Filesystem Library build, 
> which has three input sources.
>
> Here were the test cases:
>
> 1) No existing bin directories. Both build and test ran fine. As expected, 
> the residue files were a bit different: There was no path_test.success, and 
> path_test.test contained the word "passed" instead of the path to the .cpp 
> file. I've haven't looked yet to see if the lack of the path is a problem 
> for reporting, but hopefully the information is trivially available 
> somewhere else.
>
> 2) Missing source file for the library build. path_test.test was deleted, 
> so the reporting programs would know that failure occurred. The stdout 
> messages also indicated what had happened. Excellent!
>
> 3) Missing file restored. Worked fine; path_test.test was recreated, no 
> other files were touched.
>
> 4) Introduced error into one of the library files, causing a library build 
> compile to fail. path_test.test was deleted, so the reporting programs 
> would know that failure occurred. Excellent! This is the case that has 
> caused regression testing to report the wrong results in the past, so it 
> was good news to see it working correctly now. We probably should figure 
> out some other test cases just to be sure it is working for full coverage.
>
> 5) Fixed the error in the library file. The library build then worked, and 
> path_test.exe was relinked, without first recompiling path_test.obj. Test 
> was rerun. Exactly right behavior!
>
> 6) Ran bjam again, w/o changing any dependencies. As hoped, nothing was 
> out of date, so no actions taken.
>
> Because of non-Boost time pressures, it will be the end of next week before 
> I can modify the status reporting programs to use "sane-testing".
>
> Thanks for the improvements!

-- 
David Abrahams
dave_at_[hidden] * http://www.boost-consulting.com
Boost support, enhancements, training, and commercial distribution