Re: [boost] What Should we do About Boost.Test?

Subject: Re: [boost] What Should we do About Boost.Test?
From: John Maddock (boost.regex_at_[hidden])
Date: 2012-09-17 13:02:45

> I don't know what to do about this.  Because of the lack of redundancy
> (i.e. tests and documentation), it's hard to tell whether this library
> is correct or even to define what "correct" should mean.  It seems like,
> as long as the code is incompletely / incorrectly documented and tested,
> it's just someone's personal coding project that we happen to keep
> shipping with Boost, and not really a library for general use.  This
> situation reflects poorly on Boost as a whole and the fact that it
> centers around a _testing_ library, which is concerned with
> robustness...  well, let's just say that the irony isn't lost on me.

While we're on the subject, I confess I thought about writing something in 
this area, but hey, too much already! :-)

Here's my wish list though:

* Lightweight, header only if possible.
* Clear separation between components (execution monitor, from unit test 
call framework, from testing macros).  Ideally each would be a separate mini 
library if that's possible, with the executable linking against just what it 
needs and no more.
* Thread safe from the start, testing from multiple threads should be a 
no-brainer.
* Easy debugging: if I step into a test case in the debugger the first thing 
I should see is *my code*.  As it is I have to step in and out of dozens of 
Boost.Test functions before I get to my code.  This one really annoys me.
* Rapid execution of each test case, a BOOST_CHECK(no-op) should be as near 
to a no-op as possible.  I was unable to use Boost.Test for a lot of the 
Math lib tests for this reason - looping over thousands of tests was simply 
impractical from a time point of view (maybe this has improved since then, I 
haven't checked).
* Exemplary error messages when things fail - Boost.Test has improved in 
this area, but IMO not enough.
* An easy way to tell if the last test has failed, and/or an easy way to 
print auxiliary information when the last test has failed.  This is 
primarily for testing in loops, when iterating over tabulated test data.
* Relatively simple C++ code, with no advanced/poorly supported compiler 
features used.  This is one library that should be usable anywhere and 
everywhere.
* Ultra stable code.  Exempting bug fixes, I'd like to see a testing library 
almost never change, or only change after very careful consideration, for 
example if a new C++ language feature requires special testing support.

And what I don't want:

* Breaking changes: Boost authors have absolutely no time to track breaking 
changes in their dependencies, since a successful testing library would be 
used universally by all of Boost, this is particularly important for this 
library.
* No feature creep.  Keep it small, focused, quick to compile.  If new 
features are added they should be separate (i.e. only pay for what you use).

John.