[boost] [test] Support use of test tools and report generation in non-test programs - facilitating test/validation modes

Subject: [boost] [test] Support use of test tools and report generation in non-test programs - facilitating test/validation modes
From: Jamie Allsop (ja11sop_at_[hidden])
Date: 2012-09-26 07:48:28

An important use case that we have is the ability to run test scenarios 
on a production binary by specifying a 'test' mode. In test mode we 
essentially run through a series of different scenarios, asserting 
various conditions, comparing expected results and so on. In order to 
minimise the custom code we needed for this we decided to make use of 
Boost.Test. All the test tools are there, report generation and so on. 
In addition we have all the report analysis tools in place to interpret 
the output.

The problem however, is that Boost.Test essentially requires a custom 
main() function. Either one of the ones provided by Boost.Test or one 
performing Boost.Test specific initialisation. In our production systems 
we have, like many, a fairly standard main() that ensures uniform 
behaviour across binaries and we do not want to pollute that with 
Boost.Test specific code. Notably we also do not want to pollute our 
command line parameters or configuration with 'global' Boost.Test 
arguments to control the test report output. Finally we want to control 
exactly when the tests are executed.

I've created ticket #7419 to cover this use case and have also attached 
a small patch that facilitates the capabilities required to support it.
https://svn.boost.org/trac/boost/ticket/7419

Essentially the patch does two things:

   * It allows boost::unit_test_framework::init() to be called more than 
once. The result is to reset the test status and master test suite so 
that new tests can be added and new reports generated through multiple 
calls. The CLA parsing is also reset. Recent changes to this code may 
make my call to clear() redundant. I'll let Gennadiy be the judge of that.
   * This first change then allows the framework setup and teardown to 
be encapsulated in a helper class (not provided in the patch but one 
could be). In our systems we use a helper class that exposes a simple 
API that allows the user to implement a method where test units can be 
added directly to the master test suite - see the ticket for an example. 
As I said the patch focuses solely on enabling the possibility of 
writing such a class, not the class itself.
   * It provides scoped Boost.Test arguments for the commandline when 
BOOST_TEST_USE_QUALIFIED_COMMANDLINE_ARGUMENTS is defined. All command 
line arguments are then called --boost.test.<argument> for example 
--boost.test.log_level instead of --log_level. This of course means that 
all Boost.Test related arguments will not clash with any program 
arguments. On its own this is a generally useful feature as it allows 
custom main() functions to pass arguments down to other components 
during simple unit testing.

The patch attached is based on a patch against boost 1.49. The one 
attached to the ticket is for trunk. Again this is a small patch that is 
genuinely useful so I'd like to see it make its way into trunk. As far 
as I know there are no side-effects for adding these changes, tests all 
pass and so on.

Jamie