Re: [jamboost] Re: Sane regression testing

From: Beman Dawes (bdawes_at_[hidden])
Date: 2002-11-06 10:47:35

At 08:48 PM 11/5/2002, David Abrahams wrote:

>> 1) No existing bin directories. Both build and test ran fine. As
>> expected,
>> the residue files were a bit different: There was no path_test.success, 

>> and
>> path_test.test contained the word "passed" instead of the path to
>> the .cpp
>> file. I've haven't looked yet to see if the lack of the path is a
>> problem
>> for reporting, but hopefully the information is trivially available
>> somewhere else.
>
>If you want that info, I can put it in there. Please confirm that you
>want it if so, and tell me if there's anything else you want to see in
>the file.

If it is easy, please do add it as the first line of the .test file. But 
don't spend more than say 30 minutes on it as I can synthesize the 
information if it is a problem for you.

>
>> 6) Ran bjam again, w/o changing any dependencies. As hoped, nothing 
was
>> out of date, so no actions taken.
>
>I can try to encapsulate this list of actions and expected results in
>one of our Python testing scripts. Please see various scripts in
>tools/build/test/*.py for examples. Specifically,
>testing_primitives.py shows how I tested the basic components of the
>new test system. I think these scripts are fairly self-explanatory,
>and after I set up the cases above you should be able to add your own
>without difficulty.

Sounds interesting. I'd particularly like to be sure case (4) (previously 
OK test fails when lib it depends on now has compile failure) is regression 
tested as it happens a lot in practice. Also, I might be able to use your 
test scripts as tests for status reporting programs. I don't have a good 
way to test them now; just running them only shows if they work for the 
current state of the Boost CVS, which often doesn't include various corner 
cases.

--Beman