[boost] Boost.Test: major update

From: Gennadiy Rozental (gennadiy.rozental_at_[hidden])
Date: 2005-05-08 03:46:07

Hi,

I would like to take this moment to share with you finally the modifications 
Boost.Test went through during last couple month. Most of Boost.Test 
components undergo a major or significant change in both design and 
implementation. I've strived to keep an interface unchanged as much as 
possible. The primary goals for this update were:
  * Make a unit test framework better suited for different 'runners'
  * Make unit test framework simpler internally by eliminating test cases 
hierarchy
  * Make test tools more uniform
  * Make auto unit test framework as usable as possible

Following are details of what updated/changed specifically in each area of 
Boost.Test:

I. Execution Monitor

The execution monitor design changes from inheritance to delegation. If 
previously one needed to inherit from the boost::execution monitor to do a 
monitored run of a function, now one need to employ an instance of the 
boost::execution_monitor:

boost::execution_monitor em;

// register translators for custom user exception
em.register_exception_translator<ExceptionType>( my_translator );

em.execute( function_to_execute );

function_to_execute should be any zero arity function or function object. To 
implement the polymorphic function handling I employ my own class callback 
(see utilities section). I did not want to introduce a dependency on 
Boost.Function. But in a future I plan to switch to tr1::function.

In addition following features implemented:

* Debugger presence detected, in which case catching system errors (SEH) is 
disabled unless explicitly enabled. For now only works for msvc family
* Automatic detection of memory leaks introduced. For now only works for 
msvc family
   Unit test framework turns this feature on by default. User could manage 
it by specifying runtime parameter (in a form of command line argument or 
environment variable). For example

test --detect_memory_leak=<leak_allocation_number>

set leak_allocation_number to integer number:
0 - to disable detection of memory leaks;
1 - to enable detection of memory leaks;
N>1 - to enable detection of memory leaks and set a system breakpoint at the 
moment of Nth memory allocation. For more on how to find N and some issues 
with usage of this feature see documentation

II Unit test framework (UTF)

  This component of Boost.Test undergo a most dramatic changes in both 
design and implementation. Here are main modifications:
  1. UTF doesn't employ test cases hierarchy anymore. There is single class 
test_case that model one testing unit and class test_suite that model a 
collection of test units. Previously UTF employed compound test cases (for 
example parameterized test case) which went through it's components (simple 
subtests) during testing run time. Now compound test present itself as test 
case generator which produce a collection if test_case instances during test 
tree construction time. One consequence of this is that user doesn't need to 
keep track of compound test case parameters is it used be. test_case now 
employs a polymorphic callback to a actual test function. As a result of 
this one could supply any zero arity function as an argument of 
BOOST_TEST_CASE. SO no extra support needed to be able to create 
boost::function or boost::bind based tests cases.

   2 .Introduced generic mechanism to traverse the test tree. Anyone willing 
to do so just need to implement test_tree_visitor interface and employ 
traverse_test_tree. This allows to implement any pre/post processing over 
test tree.

   3. Introduced a notion of framework. The framework is responcible for:
      a) initializing and providing access to master test suite
      b) providing access to current test case
      c) running testing from given test tree node (simple or compound). By 
default it runs from root - master test suite. Supports both sequential and 
random order.
      d) One may register any test observer agent (implements test_observer 
interface) and framework notifies it of all test events
      e) providing access to any test case based to unique test unit id

Using the framework one could implement different test runners. UTF library 
supplies one console runner (unit_test_main.cpp). But it's possible to do 
one winmain based or anything else. For example GUI program that build test 
tree and then run tests by request from any point in test tree.

   4. Unit test log interface changed to single line (used to be begin/end 
macros). Unit test log formatter interface changed. How it much simpler and 
straightforward. Two supplied implementation are separated.

   5. Notion of unit test result that was responcible for result collection 
and reporting eliminated. Instead introduced independent notions: 
results_collector and results_reporter. Results reporting formatter 
interface made public. Also results_reporter now allows to change an output 
stream.

   6. Progress monitor functionality (which used to be activated in log 
level log_progress) now is an independent test observer and implemented by 
class progress_monitor.

   7. Test case template interface changed. Now user doesn't need to 
explicitly register meta class.

In addition couple more features implemented:
   * Added support for test case timing
   * UTF made __cdecl - safe
   * Result report include skipped tests
   * io saver facilities employed to guard against undesirable modification 
in output streams formatting
   * template test case automatically generate name that includes type name
   * --detect_memory_leak and --random cla added

III Test Tools

The two major changes in Test Tools are:

   * Test Tools interfaces to provide 3 version for each tool:
       CHECK- reports an error if assertion failed
       WARN - reports a warning if assertion failed
       REQUIRE - reports an error if assertion failed and aborts test case 
execution
   * Test Tools implementation completely reworked. All tools implemented 
now through single vararg function

In addition following modifications made:
   * CHECK_COLLECTION interface changed: it now expect 4 arguments
   * BITWISE_EQUAL renamed to CHECK_BITWISE_EQUAL; old name still provided, 
but is deprecated
   * CHECK_COLLECTION interface changed to use PP_SEQ and as a result 
support arbitrary(actually maximum is fixed but configurable) number of 
predicate arguments
   * most of templates eliminated speeding up test modules compilation
   * floating-point precision set to include all significant digits 
(actually the formula may require some rework since it obviously doesn't do 
a good job sometimes)
   * BOOST_CHECK/WARN/REQUIRE_SMALL test tool added
   * old test tools names are *deprecated*

IV Auto unit test framework

Interesting discussion with Noel Llopis and several user requests lead to 
several additions to the auto_unit_test.hpp. Here is a short descriptions:

   * BOOST_AUTO_UNIT_TEST renamed to BOOST_AUTO_TEST_CASE. Old name still 
provided but deprecated
   * BOOST_AUTO_TEST_CASE_TEMPLATE provides an ability to write a 
self-registering test case template
   * BOOST_FIXTURE_TEST_CASE provides an ability to write a self-registering 
test case with fixture
   * BOOST_AUTO_TEST_CASE_EXPECTED_FAILURES provides an ability to specify a 
number of expected failures in any self-registering test case
   * BOOST_AUTO_TEST_SUITE and BOOST_AUTO_TEST_SUITE_END provide an ability 
to structure test cases into self-registering test suites

V Included components

Implementation moved from source files (.cpp) under lib/test/src into 
implementation files (.ipp) under boost/test/impl to eliminate dependency of 
included/minimal components on lib/test/src directory.

VI Utilities

Basic utilities that may have value outside of Boost.Test now separated into 
utils subdirectory. If there is an interest to promote any of them onto 
public boost level I am ready to discuss it. Here is a short descriptions 
(some of them existed before, but most are new):

   1) algorithm.hpp
   Implements several variations of STL algorithms including 4-arg mismatch

   2) assign_op.hpp
   Hook for overloadable assignment

   3) callback.hpp
   Implements a polymorphic function callback facility (similar to 
boost::function).

   4) class_properties.hpp
   Simple facility that mimic notion of read-only read-write properties in 
C++ classes

   5) custom_manip.hpp
   Simple helpers for creating custom output manipulators

   6) fixed_mapping.hpp
   Fixed sized mapping (initialized in constructor ) with specified invalid 
value.

   7) foreach.hpp
   This is an abridged version of BOOST_FOREACH. The only difference is 
portability.

   8) named_params.hpp
   Facility for named function parameters support. Even though Dave A. did 
not see a value in my suggestion during Boost.Parameter review period. I 
couldn't justify for myself using that inconvenient interface (IMO) and 
absence of optional parameters. This facility implements an alternative 
solution with different set of tradeoffs and features. It used in several 
places in Boost.Test.

   9) nullstream.hpp
   I should probably deprecate this one , but more-io was not accepted.

   10) rtti.hpp
   Simple alternative to compiler RTTI. Provides an access to unique type id 
at runtime. I plan to extend this to implement type_descr similar to what is 
described in "C++ template metaprogramming".

   11) trivial_singleton.hpp
   As the name suggest this simple singleton helper.

   12) wrap_stringstream.hpp
    Wraps strstream and stringstream (depends with one is present) to 
provide the unified interface

   13) basic_cstring/
    class template basic_cstring wraps raw memory pointers and provide 
std_string like interface

   14) iterator/input_iterator_facade.hpp
    Helps to build input iterators

   15) iterator/token_iterator.hpp
    Yet another token iterator

   16) iterator/istream_line_iterator.hpp
    Iterate over istream and produce lines

   17) runtime/
   This is long promised framework for parsing CLA handing environment 
variable and parsing config files. This is an alternative solution to what 
boost::program_options present. Unfortunately I did not have enough time to 
incorporate it completely into Boost.Test main components, so the detailed 
description will follow once this will happened.

Sorry, for babbling for so long. Any comments are greatly appreciated.

Regard,

Gennadiy