Re: [boost] Regression tests: how long is reasonable?

From: Beman Dawes (bdawes_at_[hidden])
Date: 2002-08-12 13:26:40

At 11:58 AM 8/12/2002, Daniel Frey wrote:

 >...
 >
 >Maybe we should separate tests to pure regression tests that check the
 >existence / correctness of a feature and benchmark-tests that give
 >information about the efficiency / speed of some feature. During
 >development, you run the regression tests first and only if they pass,
 >you may want to run the benchmark tests. The latter would not report
 >results as booleans (pass/fail), but it will have timings, object
 >counters, size of objects (to see the overhead of a library), etc.

Interesting.  I already have a request in to the Jam Boost folks for 
version 2 to supply timings, but had not given any thought to space and 
size issues.

Space and size is probably not something bjam can report directly.  Some 
subsidiary process is likely to be needed to capture that information. Does 
anyone have any ideas about how that might work?

After so many people have put so much effort into the bjam regression 
testing system, and now that the compiler vendors are taking regular notice 
of the results we post, I've come to feel we should definitely report more 
than just pass/fail.

We probably won't do much more right away, since we need to finish the 
current transition to bjam based tests, and version 2 of Boost.Build isn't 
ready anyhow.  But we do need to start thinking about future directions for 
testing.  Better factoring and other organizational issues are part of it, 
but time and space testing should also be given attention, IMO.

--Beman