[boost] Regression tests: how long is reasonable?

From: Beman Dawes (bdawes_at_[hidden])
Date: 2002-08-12 10:03:15

How long is it reasonable for a regression test to run?

Tests that take a long time cause several problems:

*  They're hard on the regression testers, who often have to run tests over 
and over while diagnosing a problem.

*  Boost.Test Version 2 has the ability to "time-out" a test, under the 
assumption it is looping.  We need to know where to set that time-out 
point.

A specific example:  the operators_test run has a probe (with a random 
number) it runs 10,000 times. On the 2 GHz Pentium used to run the Win32 
tests, that takes 33 seconds (Borland) to 111 seconds (Intel).  Say an 
average of 60 seconds.  That means total time for this one test of from 
five to ten minutes, depending on how many compilers are in the mix.

I'd like to see a 5 second maximum each for both compiles and runs on the 
main regression test.

Of course, a library would be free to include compiles and runs which take 
any length of time, as long as they aren't included in the main tests.

Comments?

--Beman

PS: For now I'm dropping the operators_test probes to 1,000.