Review: Test library

From: Ed Brey (brey_at_[hidden])
Date: 2001-02-13 14:47:20

I think the Test library should be accepted into boost.  I think the
breakdown of execution, test, and unit test tools is excellent.  Comments
regarding areas for improvement:

catch_exceptions.hpp:

1. Why is BOOST_GENERAL_ERROR_RETURN made to look like a macro, and not in a
namespace?  If trying to be like EXIT_FAILURE, what is the difference?
Assuming there is a good reason, then there is the issue that the anonymous
namespace around BOOST_GENERAL_ERROR_RETURN is unnecessary; since it's a
constant it doesn't get external linkage.

2. RTTI should be used on platforms where appropriate to provide display of
precise exception class names (as pre earlier post).  When RTTI is not
appropriate, all standard exceptions should be hard coded; currently missing
is std::ios_base::failure.

3. Function_object be checked against EXIT_SUCCESS instead of zero to
determine success, unless there is a good reason to do otherwise?  A
rationale should be documented.

test_tools.hpp:

1. (void)0 could become static_cast<void>(0).  The one benefit I can see is
this allows one to run a lint check looking for C-style casts on boost
(either by itself or as a component of a larger work, which is using boost).

test_tools.cpp:

1. It might be nice to allow the output to be any stream, instead of always
cout.

test_main.cpp:

1. test_tools_exception has superfluous destructor definition.  As I recall,
a previous post asserted that a non-throwing destructor is generated by
default in a case such as this.

2. Why does BOOST_TEST_ERROR_RETURN needed rather than reusing
BOOST_GENERAL_ERROR_RETURN?  The "why make it look like a macro" question
applies here as well.

execution_tools.htm:

1. Hello world string literal has slash instead of backslash.

2. Hello world example should have argc and argv commented out to show
example of how avoid warnings when not using arguments.

test_tools.htm:

1. (This one's my favorite.)  "Boost Unit Test Tolls may be more suitable
for complex test programs."  If boost doesn't find a good funding source
through a book or journal, maybe this is the right answer.  :-).

2. Talks about "std::EXIT_SUCCESS" and "std::EXIT_FAILURE".  These names are
macros and are not in any namespace.  Also, it is implied that EXIT_SUCCESS
is always 0 and that EXIT_FAILURE is always non-zero.  The Standard doesn't
seem to guarantee such; rather it says that success is returned if and only
if exit is passed either 0 or EXIT_SUCCESS (but EXIT_SUCCESS is not
constrained to be either the same or different than 0). 18.3/8

General:

1. Capitalization is inconsistent: Only the "Errors detected; see stdout for
details" has a capital letter; nothing else does.  My preference would be to
always start with a capital letter and end with a period for canned summary
messages "Returning with error code #." and "Errors detected; see stdout for
details.", but leave detail messages (exceptions, test failed) as is.

2. Conditional compilation blocks should list standard code first, then
compiler workarounds.

3. Lost of extra spaces in docs.  When I use FrontPage, I too find it is
easy to have extra spaces sneak in.  The solution to find them is pretty
simple: grep for "nbsp".

4. I agree with a previous post that calls for giving several of the global
variables a home.

Pet feature (MSVC-specific):

Test programs do well as console (as opposed to GUI) applications, since
there are many advantages to having them be completely automated; hence no
GUI is needed for user input.  The test tools suite is very fits in very
well in this regard.  For debugging, it is often very efficient to use the
exact same console application written for automated testing, as one can
often, by quickly commented out working sections of a test, reproduce bugs
quickly in a deterministic way, and get to breakpoints very quickly.

The catch is that in the Windows environment, when using VC (and possibly
other vendors' tools), debugging a console test application can be annoying
and less productive that debugging an application that writes its output to
the Windows debug stream.  This avoids the console from popping up any time
code is running between breakpoints, and avoids having to switch to the
console window to view output (VC provides a good deal of control over where
the debug stream is displayed).

The best of both worlds is a test executable that could run as a console
application by default (useful when running tests outside a debugger), but
run as a basic Win32, non-console application (useful when debugging) with a
command line switch.  Actually achieving run-time behavior such as this
isn't trivial (at least for me; I went in knee deep and got stuck).  If
anyone else is interested in this kind of functionality, contact me, and I
can share what I know.  In general, I would like to present this use case as
something to think about (and balance against simplicity) in keeping the
test tools library flexible.