Re: [boost] Regression tests: how long is reasonable?

From: Beman Dawes (bdawes_at_[hidden])
Date: 2002-08-12 15:49:48

At 04:24 PM 8/12/2002, Gennadiy Rozental wrote:
 >>Maybe we should separate tests to pure regression tests that check the
 > >existence / correctness of a feature and benchmark-tests that give
 > >information about the efficiency / speed of some feature. During
 > >development, you run the regression tests first and only if they pass,
 > >you may want to run the benchmark tests. The latter would not report
 > >results as booleans (pass/fail), but it will have timings, object
 > >counters, size of objects (to see the overhead of a library), etc.
 >
 >1. I am planning to add test case some basic timing really soon.

That may be redundant.  Boost already has a timer library.  See 
www.boost.org/libs/timer/timer.htm

Plus, we need the ability to time execution regardless of whether on not a 
test program uses Boost.Test. Thus the bjam framework has to provide the 
time.  Compiles, links, and lib events, need to be timed too, and the bjam 
framework is the only place to do that.

 >2. On some (UNIX) systems test case timeout could be set right now. At 
some
 >point I will try to tuckle windows timeouts.

I've already got contributions of Win32 timeouts from two people. But 
nowadays, we've got Boost Threads, so we might think of a generalized 
approach.  The only trouble with that is the dependency issue; since we use 
Boost.Test to test Boost.Threads, it may not be a hot idea to make 
Boost.Test dependent on Boost.Threads:-(

 >3. One could now print object sizes using BTL BOOST_MESSAGE facility. It
 >does not affect pass/fail status - it's just the way to print an info.

Yes, that would fit into the regression test reporting system pretty 
well.  But I want to press people who output test data that they expect 
other programs to read to use XML to make the recognition and parsing easy.

 >4. I am planning to add named test case run soon. This way regression 
test
 >facility could run only validating test cases/suites.

Could you explain that a bit more?

Thanks,

--Beman