[boost] Boost.Test update: run by name, debug interfaces, new tools and more

From: Gennadiy Rozental (rogeeff_at_[hidden])
Date: 2007-10-26 03:22:54

Hi,

Last couple weeks I spent incorporating new features into Boost.Test trunk. 
Some of these were sitting on my local disk for quite some time, but it's 
finally there. There are still number of features/request in pipeline to be 
addressed, but I plan to concentrate primarily on docs update now. BTW they 
are progressing and in much better shape now. I might need to work to 
include all the fixes required to build it into boostbook. I plan to put 
beta on my site soonish. Following are more or less detailed release notes 
for all the changes. Any comments are welcome as usual.

Regards,

Gennadiy

=======================================================
Release notes:
=======================================================

I. New debug interfaces

In this update I made an effort to introduce portable interface for 
communication with debugger and other debug facilities. The interface is 
defined as set of free functions along with some helper structures. I left 
out couple functions related to the portable stack dump/stack walk 
interface, since it's not completely ready yet and I am do not have enough 
time at the moment to finish it.

New interface consists of the following functions:

---------
bool under_debugger();

Detects if program is running under debugger
---------
void debugger_break();

Causes program to break execution in debugger at call point
---------
bool attach_debugger( bool break_or_continue = true );

Attaches debugger to the current process
---------
void detect_memory_leaks( bool on_off );

Switches on/off memory leaks detection
---------
void break_memory_alloc( long mem_alloc_order_num );

Causes program to break execution in debugger at specific allocation point
---------

I've made an effort to implement these interface for number of available 
configurations/debuggers for platforms I had an access to:NT, Linux, 
Solaris. There are still a lot of work to make these interfaces available on 
other systems. These is also interfaces to add support for additional 
debuggers, setup default debuggers and configure some parameters of existing 
implementation.

>From usability standpoint there is new command line argument available for 
UTF users: --auto_start_dbg=<yes|no|debugger_name>
Default value is no. If set to yes, in case of system error (sigsegv for 
example) the program will try to attach default debugger defined for the 
toolset and stop at fault location. If set to the different value this value 
is treated as debugger id (one of the predefined set of supported debuggers) 
and Boost.Test will try to use this debugger. For 
example --auto_start_dbg=dbx-ddd causes ddd with dbx to be used, 
while --auto_start_dbg=gdb-xterm will open new xterm window and attach gdb.

Some of the functions are implemented only for specific platforms for now. 
But the main goal of this exercise is to define portable interfaces.

II. Execution monitor rework

The execution monitor implementation is rewritten almost completely. Some 
new features introduces, some old done differently.

1. windows SEH handling
Windows SEH handling changed completely. Instead of seh translator execution 
monitor relies on custom exception filters and double set of __try/__catch 
constructs. There some additional tricks in place to make it work more 
reliably.

2. Signal handling on *nix
Signal handling is made more consistent and covers more signals. An 
alternative stack is used by default  signal handler by default, while there 
is an option to avoid it (--use_alt_stack=no for UTF users)

3. New portable debug interfaces used in place of platform specific calls.

4. Error reporting is enhanced both on windows and *nix to provide more 
information about error cause and location. For example these lines
    int* p = (int*)0x01;
    BOOST_CHECK( *p == 0 );

will now produce

"fatal error in "free_test_function": memory access violation occurred at 
address 0x00000001, while attempting to  read inaccessible data"

instead of

"fatal error in "free_test_function": memory access violation"

Similar enhancements made for different types of system errors.

5. Handling for invalid parameter error produced by VC 8.0 runtime 
introduced

6. Floating point exceptions detection is introduced.
For windows based configurations you can tell the execution monitor to 
detect floating point errors like underflow,overflow,divide by zero and 
others. It's disabled by default. UTF users can enable it 
using --detect_fp_exceptions=yes

III. Significant framework updates

1. run by name is finally there

UTF users can mow specify test unit(s) to run using command  line argument. 
Use

<test-module> --run_test=<test to run>

Test to run specifies which test cases/test suites to execute. It specified 
in a form a/b/c, where a,b and c are filters for corresponding levels of 
test tree. '*' can be used at the beginning, at the end and as the level 
filter itself. ',' is used to create lists. Here couple concrete examples:

--run_test=test1 - runs test case or test suite test1 that resides in master 
test suite.
--run_test=test1,test2 - runs test case or test suite test1 and test case or 
test suite test2 that resides in master test suite.
--run_test=s1/test1 - runs test case or test suite test1 that resides in 
test suite s1 that resides in master test suite.
--run_test=s1*/test1 - runs test case or test suite test1 that resides in 
test suites with name stating with s1 and that resides in master test suite.
--run_test=*s1/test1 - runs test case or test suite test1 that resides in 
test suites with name ending with s1 and that resides in master test
--run_test=*/test1* - runs any test unit with name starting with test1 that 
resides in any test suite that resides in master test
--run_test=* - runs all test units

Note

--run_test=a/* is equivalent to --runtest=a

2. New tools in toolbox.

Responding to the several requests I've implemented 5 new tools:

BOOST_CHECK_NE(a,b)
BOOST_CHECK_LT(a,b)
BOOST_CHECK_LE(a,b)
BOOST_CHECK_GT(a,b)
BOOST_CHECK_GE(a,b)

Floating point tools update unfortunately did not make a cut.

3. Test suite auto registration modified to allow c++ namespace like 
behavior.

It's now possible to restart the test suites and defines the same test 
suites in different test file, effectively making test suites similar to the 
C++ namespaces. for example

BOOST_AUTO_TEST_SUITE(s1)

BOOST_AUTO_TEST_CASE(t1)
{
}

BOOST_AUTO_TEST_SUITE_END()

BOOST_AUTO_TEST_SUITE(s1)

BOOST_AUTO_TEST_CASE(t2)
{
}

BOOST_AUTO_TEST_SUITE_END()

construct a test tree with single test suite s1 containing 2 test cases t1 
and t2.

4. Support for expected failures in test cases with automated registration 
is reworked completely. It now allows to be used within auto-test-suites.

5. unit_test::framework API changed to return non const references to the 
test units to allow post creation modifications. It now possible to modify 
test tree at runtime. In addition test_suite::remove interface is introduced 
to allow remove test units from the test suite if necessary.

6. init_unit_test_suite function invocation is now guarded, so any 
exceptions produced by it doesn't cause UTF to report internal failure but 
show correct cause of setup failure.

VI. Minor modifications and fixed

* Use portable message mechanism in execution monitor
* Interface to access results reporter stream provided
* Bug in output_test_stream constructor error generation fixed
* Test start and finish messages disabled if log level is set to nothing
* Test module initialization error message is redirected into result 
reporter stream
* New message is added to report no assertion occurred in a test case
* Framework misuse guard is introduced to prevent usage of testing tools 
before framework is initialized (for example from within init function)