[boost] Complex testing requirements

From: William E. Kempf (wekempf_at_[hidden])
Date: 2003-01-31 04:18:14

One of the many things I'm attempting to do right now is to improve the
testing of Boost.Threads.  I'd really like to use a more complex testing
system than seems available with the current Boost tools.  Or maybe I'm
wrong, and it is possible.  Here's a description of my requirements.

* Test cases/suites need to be defined in a tree hierarchy, where branches
are never run if the parent test doesn't pass.

* These test cases may include tests for compilation and/or link
success/failure.

* By default I would like to build and test the entire tree of test
cases/suites, obeying the first rule I provided above, but there should be
a way to build and test a single test case/suite with out regard to the
parent (i.e. force a build and test of a single suite with out regard to
any others in the tree).

The reasoning behind these requirements:

* Many tests in Boost.Threads depend on portions of the library that's not
being directly tested.  For instance, to fully test mutex locks I need to
create some threads, so it's pointless to run these tests if the test for
thread creation fails.  In fact, there are cases where deadlocks could be
possible because of this, and coding around those cases is a lot of work
just for creating a test case.

* The total regression test is fairly extensive (and will be even more so
before we're done), so running the full test when debugging a single
concept would be too time consuming.  Thus the need for running a single
test case with out regard for any dependencies on other test cases (where
it's up to the user to ensure dependent portions are working).

I could manage to implement the "test dependency tree" if I were to put
all tests into a single program.  However, this doesn't allow for
compilation/linking tests, and doesn't address the ability to quickly
compile and test a single test case/suite.

I can continue on with out these capabilities, but I'd like to know if I'm
missing something that would allow me to do all of this with the tools as
they are today, or if there's enough interest in these capabilities for
them to be addressed in our testing tools in the future.

William E. Kempf
wekempf_at_[hidden]