Re: [jamboost] Re: Sane regression testing

From: David Abrahams (dave_at_[hidden])
Date: 2002-11-06 10:58:04

Beman Dawes <bdawes_at_[hidden]> writes:

> At 08:48 PM 11/5/2002, David Abrahams wrote:
> 
> 
> >> 1) No existing bin directories. Both build and test ran fine. As
> >> expected,
> >> the residue files were a bit different: There was no path_test.success, 
> 
> >> and
> >> path_test.test contained the word "passed" instead of the path to
> >> the .cpp
> >> file. I've haven't looked yet to see if the lack of the path is a
> >> problem
> >> for reporting, but hopefully the information is trivially available
> >> somewhere else.
> >
> >If you want that info, I can put it in there. Please confirm that you
> >want it if so, and tell me if there's anything else you want to see in
> >the file.
> 
> If it is easy, please do add it as the first line of the .test file. But 
> don't spend more than say 30 minutes on it as I can synthesize the 
> information if it is a problem for you.

It's easy enough, but will it really be OK for you? The .test file
disappears when a test fails, so you won't have the info in that
case. Maybe you'd like some other information instead?

> >I can try to encapsulate this list of actions and expected results in
> >one of our Python testing scripts. Please see various scripts in
> >tools/build/test/*.py for examples. Specifically,
> >testing_primitives.py shows how I tested the basic components of the
> >new test system. I think these scripts are fairly self-explanatory,
> >and after I set up the cases above you should be able to add your own
> >without difficulty.
> 
> Sounds interesting. I'd particularly like to be sure case (4) (previously 
> OK test fails when lib it depends on now has compile failure) is regression 
> tested as it happens a lot in practice. Also, I might be able to use your 
> test scripts as tests for status reporting programs. I don't have a good 
> way to test them now; just running them only shows if they work for the 
> current state of the Boost CVS, which often doesn't include various corner 
> cases.

I'll try to set these up later today. If I don't get to it, bug me
again later.

-- 
David Abrahams
dave_at_[hidden] * http://www.boost-consulting.com