Re: [boost] What Should we do About Boost.Test?

Subject: Re: [boost] What Should we do About Boost.Test?
From: Gennadiy Rozental (rogeeff_at_[hidden])
Date: 2012-09-17 13:26:11

John Maddock <boost.regex <at> virgin.net> writes:
> Here's my wish list though:
> 
> * Lightweight, header only if possible.

IMO, unless you are after something very trivial, there is no reason for that. 
Having prebuilt library does not make it heavyweight. More important is user
experience.

> * Clear separation between components (execution monitor, from unit test 
> call framework, from testing macros).  Ideally each would be a separate mini 

Isn't it already the case?

> library if that's possible, with the executable linking against just what it 
> needs and no more.

Isn'r it already the case?

> * Thread safe from the start, testing from multiple threads should be a 
> no-brainer.

With C++11 threading support it couple be easily done in Boost.Test. Previously
I did not want to introduce boost.thread dependency. This is in my plan

> * Easy debugging: if I step into a test case in the debugger the first thing 
> I should see is *my code*.  As it is I have to step in and out of dozens of 
> Boost.Test functions before I get to my code.  This one really annoys me.

I am not sure I follow. Setup break point in your test case it will stop right
there. Do you setup break point in first line of main? Or you mean something
completely different?

> * Rapid execution of each test case, a BOOST_CHECK(no-op) should be as near 
> to a no-op as possible.  I was unable to use Boost.Test for a lot of the 
> Math lib tests for this reason - looping over thousands of tests was simply 
> impractical from a time point of view (maybe this has improved since then, I 
> haven't checked).

BOOST_CHECK is noop (well to some degree). There is some overhead probably.
There was indeed improvement couple years ago and now we only do minimal amount
of work necessary to pass context information about.

> * Exemplary error messages when things fail - Boost.Test has improved in 
> this area, but IMO not enough.

Specifically?

> * An easy way to tell if the last test has failed, and/or an easy way to 
> print auxiliary information when the last test has failed.  This is 
> primarily for testing in loops, when iterating over tabulated test data.

This is addressed with trunk improvements. There is several tools introduced to
help with context specification.

> * Relatively simple C++ code, with no advanced/poorly supported compiler 
> features used.  This is one library that should be usable anywhere and 
> everywhere.

I do not believe Boost.Test uses in it's core any advanced C++ features. I am
looking to add new component which might use one, but it is always going to be
an extension.

> * Ultra stable code.  Exempting bug fixes, I'd like to see a testing library 
> almost never change, or only change after very careful consideration, for 
> example if a new C++ language feature requires special testing support.

Test library as any other library has users, bugs, feature requests etc. It has
a life on it's own. It does indeed need to be more carefully maintained in
comparison with other libs, but:

1. Proper component dependency helps. Your library needs to be built against
released version of Test library (even trunk one). This way Test library can do
it's own development in parallel.

2. Test library might need to be released not that frequently (that's why I
actually holding on to releasing my changes, cause there is still a chance
something somewhere will break)

3. There should be a period of time (short one) when testing library in release
branch is updated. If there are few regressions/conflicts these can be fixed.
Otherwise the change is reverted.

> And what I don't want:
> 
> * Breaking changes: Boost authors have absolutely no time to track breaking 
> changes in their dependencies, since a successful testing library would be 
> used universally by all of Boost, this is particularly important for this 
> library.

Again: proper component dependency. Depending on trunk version of your
dependencies is the root cause of the issue here. One library should depend on
specific released version of another library A.deps = B:1.2.3

> * No feature creep.  Keep it small, focused, quick to compile.  If new 
> features are added they should be separate (i.e. only pay for what you use).

This is mostly the case. Do you have any examples otherwise?

Gennadiy