Re: [boost] Re: How do regressions tests work?

From: Stefan Seefeld (seefeld_at_[hidden])
Date: 2003-11-19 15:13:41

David Abrahams wrote:
> Stefan Seefeld <seefeld_at_[hidden]> writes:
> 
> 
>>David Abrahams wrote:
>>
>>>Stefan Seefeld <seefeld_at_[hidden]> writes:
>>>
>>>
>>>>>It might be still possible to reuse the test running/result storing
>>>>>infrastructure, but IMO it's non-trivial project.
>>>>
>>>>I think it's well worth the efford (and not that hard). If anybody wants
>>>>to try, I'd be willing to help (though you may prefer to take the
>>>>discussion off of this list).
>>>
>>>And what's the point of this exercise?  I don't think we should
>>>introduce yet another tool which even fewer people know how to
>>>maintain unless the benefits are really dramatic.
>>
>>someone asked how tests were performed, specifically, whether there's
>>a way to compare a test's output with some expected string.
>>You answered that this was currently not possible.
>>
>>I suggested to look at qmtest, because it provides a flexible way to
>>define tests (way beyond just comparing stdout).
> 
> 
> What are the benefits it would add "beyond just comparing stdout"?

as to test types:

* you may run tests for which the evaluation of success requires more
   than 'diff', for example if you run multi-threaded or have some other
   form of 'uncertainty'.

* running the tests may require some resources to set up (databases
   the test will access, servers the test will communicate with, etc.)
   qmtest permits to define resources tests may depend on.

as to running the tests:

* qmtest is robust and portable. It can (portably) spawn subprocesses
   executing the tests, which includes the handling of failures such as
   blocking tests (i.e. it can kill the subprocesses after n seconds).

* you can set up test runs in a very flexible way, including
   single/multi-threaded, single-multi-process, local or rsh based.

results:

* qmtest maintains a 'result database', which can be a file, a
   database, etc. With that it's easy to track regressions or to set
   up expectations.

Regards,
                Stefan