Re: [boost] Regression tests: how long is reasonable?

From: Daniel Frey (daniel.frey_at_[hidden])
Date: 2002-08-12 10:58:33

Beman Dawes wrote:
> 
> How long is it reasonable for a regression test to run?
> 
> Tests that take a long time cause several problems:
> 
> *  They're hard on the regression testers, who often have to run tests over
> and over while diagnosing a problem.
> 
> *  Boost.Test Version 2 has the ability to "time-out" a test, under the
> assumption it is looping.  We need to know where to set that time-out
> point.
> 
> A specific example:  the operators_test run has a probe (with a random
> number) it runs 10,000 times. On the 2 GHz Pentium used to run the Win32
> tests, that takes 33 seconds (Borland) to 111 seconds (Intel).  Say an
> average of 60 seconds.  That means total time for this one test of from
> five to ten minutes, depending on how many compilers are in the mix.
> 
> I'd like to see a 5 second maximum each for both compiles and runs on the
> main regression test.

Maybe we should separate tests to pure regression tests that check the
existence / correctness of a feature and benchmark-tests that give
information about the efficiency / speed of some feature. During
development, you run the regression tests first and only if they pass,
you may want to run the benchmark tests. The latter would not report
results as booleans (pass/fail), but it will have timings, object
counters, size of objects (to see the overhead of a library), etc.

Regards, Daniel

--
Daniel Frey
aixigo AG - financial training, research and technology
Schloß-Rahe-Straße 15, 52072 Aachen, Germany
fon: [phone], fax: [phone]
eMail: daniel.frey_at_[hidden], web: http://www.aixigo.de