[boost] Regression testing -- some new testing options?

From: Jeff Garland (jeff_at_[hidden])
Date: 2004-05-09 18:55:39

Robert Ramey & I have been having a discussion offline about the impact of new
boost libraries on the time it takes to run the regression tests. 
Serialization has a particularly time consuming test -- the so called 'torture
test' suite.  The problem is, that once serialization is in the tree it will
be important to run that full test from time to time (like right before
release), but it would probably be overkill to run everyday.  And from past
dicussion it is clear that running the regression test is already becoming
very time consuming. 

Of course this isn't a serialization specific issue. With more and more
libraries added the time to do any sort of regression test is continuing to
grow. In addition, there is now the 'dll' versus 'static' library dimension
for those of us with compiled libraries.  It might be nice to sometimes run
one or the other to speed things up. Also, Robert would like to be able to run
'debug' versus 'release' tests from time to time.

So here's a suggestion.  Perhaps it be possible to create a standard way we
could set up the regression test targets so that library authors could:

1) define a couple standard test levels - say 'basic' and 'torture'
2) DLL versus static tests would be an option on the regression suite -- so
you could run only one or both.  Of course this isn't an issue for header
libraries.
3) debug versus release is an option
4) define specific compiler/platform exclusions

I believe such a design could increase the regression test effectiveness by
allowing some contributors and users to run only one set of options (eg:
'basic-dll-release') tests.  This option would be much-much faster than
'torture-all-all' -- which for a compiled library needs to compile and excute
4 versions of the tests (dll-release, dll-debug, static-release, static-debug).

The compiler/platform exclusions would save time by just skipping over stuff
that is going to fail and will never be fixed.  There may be some way of doing
this now, but ideally we would have some way of specifying this in the bjam
file for the tests.

Thoughts?

Jeff