Re: [boost] Re: Regression tests: how long is reasonable?

From: Beman Dawes (bdawes_at_[hidden])
Date: 2002-08-12 15:37:23

At 03:35 PM 8/12/2002, Daniel Frey wrote:
 >On Mon, 12 Aug 2002 20:26:40 +0200, Beman Dawes wrote:
 >
 >> At 11:58 AM 8/12/2002, Daniel Frey wrote:
 >>
 >>  >...
 >>  >
 >>  >Maybe we should separate tests to pure regression tests that check 
the
 >>  >existence / correctness of a feature and benchmark-tests that give
 >>  >information about the efficiency / speed of some feature. During
 >>  >development, you run the regression tests first and only if they 
pass,
 >>  >you may want to run the benchmark tests. The latter would not report
 >>  >results as booleans (pass/fail), but it will have timings, object
 >>  >counters, size of objects (to see the overhead of a library), etc.
 >>
 >> Interesting.  I already have a request in to the Jam Boost folks for
 >> version 2 to supply timings, but had not given any thought to space and
 >> size issues.
 >
 >I was thinking about my recent proposed changes to operators.hpp, where
 >the timing is only one important thing, but the size of the classes that
 >derive from operator-templates and the number of objects created (reduced
 >by the NRVO) is important, too.
 >
 >To export a values, I can only see one portable and reasonable way: Parse
 >the output. We can probably build a class that all tests have to derive
 >from, so the base class can guarantee a defined output for all tests and
 >probably some judging of it. For example, it could compare the number of
 >objects created to a supplied theoretic minimum.

Yes, there will have to be some cooperation between the test program and 
the testing framework.  That's nothing new, at least for users of 
Boost.Test.  Right now the results reported back from the test code to the 
test framework are just the return code, but we can probably define other 
results that get written to the output in an agreed upon form so that 
post-test processes can easily recover the results.

The trick would be to do it in a way that automates the introduction into 
status tables.

For example, say the boost::shared_ptr test program wrote this to cout:

     <metric type="space" description="shared_ptr<> size"
       unit="bytes" value="8"></metric>

Then reporting programs which were interested in could include these 
results in their output.  No cost to test programs which don't wish to 
report any size, no cost to reporting programs which don't care to report 
size metrics. No manual intervention.

--Beman