Re: [boost] What Should we do About Boost.Test?

Subject: Re: [boost] What Should we do About Boost.Test?
From: John Maddock (boost.regex_at_[hidden])
Date: 2012-09-18 08:18:40

>> * Clear separation between components (execution monitor, from unit test
>> call framework, from testing macros).  Ideally each would be a separate 
>> mini
>
> Isn't it already the case?

There's duplication of source between the different component libraries 
(execution monitor, test monitor, unit test).

Plus the headers seem to pull in a whole lot of stuff I never use ;-)

>> library if that's possible, with the executable linking against just what 
>> it
>> needs and no more.
>
> Isn'r it already the case?

My gut feeling is that recent releases have got slower to compile and 
#include.

>> * Easy debugging: if I step into a test case in the debugger the first 
>> thing
>> I should see is *my code*.  As it is I have to step in and out of dozens 
>> of
>> Boost.Test functions before I get to my code.  This one really annoys me.
>
> I am not sure I follow. Setup break point in your test case it will stop 
> right
> there. Do you setup break point in first line of main? Or you mean 
> something
> completely different?

No I mean that if I break on a test case and then hit "step into" in the 
debugger I have to step through your code before I get to mine, so for 
example if I break on a BOOST_CHECK_CLOSE_FRACTION and then step, I hit:

scrap.exe!boost::unit_test::basic_cstring<char const >::basic_cstring<char 
const >()  Line 163 C++

So I return from that, and step again and hit:

scrap.exe!boost::unit_test::basic_cstring<char const >::basic_cstring<char 
const >(const char * s, unsigned int arg_size)  Line 192 C++

Return and step and hit:

scrap.exe!boost::unit_test::unit_test_log_t::set_checkpoint(boost::unit_test::basic_cstring<char 
const > file, unsigned int line_num, boost::unit_test::basic_cstring<char 
const > msg)  Line 251 C++

Return and step and hit:

scrap.exe!boost::unit_test::basic_cstring<char const >::basic_cstring<char 
const >(const char * s, unsigned int arg_size)  Line 192 C++

Return and step and hit:

scrap.exe!boost::unit_test::lazy_ostream::instance()  Line 39 C++

Return and step and hit:

scrap.exe!boost::unit_test::operator<<<char const [1]>(const 
boost::unit_test::lazy_ostream & prev, const char [1]& v)  Line 83 C++

Return and step and hit:

scrap.exe!boost::test_tools::tt_detail::check_frwd<boost::test_tools::check_is_close_t,double,double,double>(boost::test_tools::check_is_close_t 
P, const boost::unit_test::lazy_ostream & assertion_descr, 
boost::unit_test::basic_cstring<char const > file_name, unsigned int 
line_num, boost::test_tools::tt_detail::tool_level tl, 
boost::test_tools::tt_detail::check_type ct, const double & arg0, const char 
* arg0_descr, const double & arg1, const char * arg1_descr, const double & 
arg2, const char * arg2_descr)  Line 293 C++

Return and step - and finally hit my code!

So I have to step through 7 of your functions before I can finally debug a 
failing test case.

>> * Rapid execution of each test case, a BOOST_CHECK(no-op) should be as 
>> near
>> to a no-op as possible.  I was unable to use Boost.Test for a lot of the
>> Math lib tests for this reason - looping over thousands of tests was 
>> simply
>> impractical from a time point of view (maybe this has improved since 
>> then, I
>> haven't checked).
>
> BOOST_CHECK is noop (well to some degree). There is some overhead 
> probably.
> There was indeed improvement couple years ago and now we only do minimal 
> amount
> of work necessary to pass context information about.

OK, I rechecked this, and BOOST_CHECK_CLOSE_FRACTION appears to have next to 
no overhead now - excellent!

>> * Exemplary error messages when things fail - Boost.Test has improved in
>> this area, but IMO not enough.
>
> Specifically?

Testing:

double a = 1;
double b = 2;
BOOST_CHECK_CLOSE_FRACTION(a, b, 0.0);

Yields:

m:/data/boost/trunk/ide/libraries/scrap/scrap.cpp(41): error: in 
"test_main_caller( argc, argv )": difference{0} between a{1} and b{2} 
exceeds 0

Leaving aside the obvious bug (the difference is not zero!!), I would have 
printed this as something like:

m:/data/boost/trunk/ide/libraries/scrap/scrap.cpp(41): error: in 
"test_main_caller( argc, argv )": difference between a and b exceeds 
specified tolerance with:
   a                  =  1.0
   b                  = 2.0
   tolerance      = 0.0
   difference     = 1.0

Which I'm sure will get mangled in email, but the idea is that the values 
are pretty printed so they all line up nicely - makes it much easier to see 
the problem compared to dumping them all on one line.

>> * An easy way to tell if the last test has failed, and/or an easy way to
>> print auxiliary information when the last test has failed.  This is
>> primarily for testing in loops, when iterating over tabulated test data.
>
> This is addressed with trunk improvements. There is several tools 
> introduced to
> help with context specification.

Such as?  Docs?

>> * Relatively simple C++ code, with no advanced/poorly supported compiler
>> features used.  This is one library that should be usable anywhere and
>> everywhere.
>
> I do not believe Boost.Test uses in it's core any advanced C++ features. I 
> am
> looking to add new component which might use one, but it is always going 
> to be
> an extension.
>
>> * Ultra stable code.  Exempting bug fixes, I'd like to see a testing 
>> library
>> almost never change, or only change after very careful consideration, for
>> example if a new C++ language feature requires special testing support.
>
> Test library as any other library has users, bugs, feature requests etc. 
> It has
> a life on it's own. It does indeed need to be more carefully maintained in
> comparison with other libs, but:
>
> 1. Proper component dependency helps. Your library needs to be built 
> against
> released version of Test library (even trunk one). This way Test library 
> can do
> it's own development in parallel.

The way Boost works at present is that if Trunk breaks, then so does my 
stuff in Trunk: as a result I can no longer see whether I'm able to merge to 
release or not if Boost.Test is broken on Trunk.  Unfortunately this has 
happened to me a few times now.

> 2. Test library might need to be released not that frequently (that's why 
> I
> actually holding on to releasing my changes, cause there is still a chance
> something somewhere will break)
>
> 3. There should be a period of time (short one) when testing library in 
> release
> branch is updated. If there are few regressions/conflicts these can be 
> fixed.
> Otherwise the change is reverted.
>
>> And what I don't want:
>>
>> * Breaking changes: Boost authors have absolutely no time to track 
>> breaking
>> changes in their dependencies, since a successful testing library would 
>> be
>> used universally by all of Boost, this is particularly important for this
>> library.
>
> Again: proper component dependency. Depending on trunk version of your
> dependencies is the root cause of the issue here. One library should 
> depend on
> specific released version of another library A.deps = B:1.2.3

And again that's not how Boost testing currently works, no matter what you 
may wish.

The issue here is that breaking changes should not be made to trunk without 
checking to see what else in Boost is depending on those features.  As you 
know that didn't happen with the last major commit.

Regards, John.