[boost] Re: How do regressions tests work?

From: David Abrahams (dave_at_[hidden])
Date: 2003-11-19 20:22:27

Stefan Seefeld <seefeld_at_[hidden]> writes:

> David Abrahams wrote:
>> Stefan Seefeld <seefeld_at_[hidden]> writes:

>>>I suggested to look at qmtest, because it provides a flexible way to
>>>define tests (way beyond just comparing stdout).
>> What are the benefits it would add "beyond just comparing stdout"?
>
> as to test types:
>
> * you may run tests for which the evaluation of success requires more
>    than 'diff', for example if you run multi-threaded or have some other
>    form of 'uncertainty'.
>
> * running the tests may require some resources to set up (databases
>    the test will access, servers the test will communicate with, etc.)
>    qmtest permits to define resources tests may depend on.
>
> as to running the tests:
>
> * qmtest is robust and portable. It can (portably) spawn subprocesses
>    executing the tests, which includes the handling of failures such as
>    blocking tests (i.e. it can kill the subprocesses after n seconds).
>
> * you can set up test runs in a very flexible way, including
>    single/multi-threaded, single-multi-process, local or rsh based.
>
> results:
>
> * qmtest maintains a 'result database', which can be a file, a
>    database, etc. With that it's easy to track regressions or to set
>    up expectations.

Thanks, that explains a lot.

-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com