Re: [boost] What Should we do About Boost.Test?

Subject: Re: [boost] What Should we do About Boost.Test?
From: Gennadiy Rozental (rogeeff_at_[hidden])
Date: 2012-09-18 13:49:04

John Maddock <boost.regex <at> virgin.net> writes:

> 
> >> * Clear separation between components (execution monitor, from unit test
> >> call framework, from testing macros).  Ideally each would be a separate 
> >> mini
> >
> > Isn't it already the case?
> 
> There's duplication of source between the different component libraries 
> (execution monitor, test monitor, unit test).

Really? can you give an example (test monitor is not supported anymore so it is
out of the picture)
 
> Plus the headers seem to pull in a whole lot of stuff I never use 

For example? 

Dependency is interesting thing. On every one developer asking for for less
dependency you'll see five asking to better user experience, so that they can
only include single header and be done with this.
 
> >> library if that's possible, with the executable linking against just what 
> >> it
> >> needs and no more.
> >
> > Isn'r it already the case?
> 
> My gut feeling is that recent releases have got slower to compile and 
> #include.

In "recent" releases Boost.Test did not change at all. In fact because of your
complains I did not release it for like 3 years now (waiting for either boost
moving to modularized setup or collecting all the changes in trunk so I can
release them on one go and deal with some failures which might happened once).
 
> >> * Easy debugging: if I step into a test case in the debugger the first 
> >> thing
> >> I should see is *my code*.  As it is I have to step in and out of dozens 
> >> of
> >> Boost.Test functions before I get to my code.  This one really annoys me.
> >
> > I am not sure I follow. Setup break point in your test case it will stop 
> > right
> > there. Do you setup break point in first line of main? Or you mean 
> > something
> > completely different?
> 
> No I mean that if I break on a test case and then hit "step into" in the 
> debugger I have to step through your code before I get to mine, so for 
> example if I break on a BOOST_CHECK_CLOSE_FRACTION and then step, I hit:
> 
[...]
> Return and step - and finally hit my code!

1. Setup a break point in your code
2. Visual studio have a solution for avoiding stepping into the code you do not
want to (I should probably include it (file) in docs)
3. There is a valid reason for these extra code. It collects some context info
(no overhead) and makes sure the code under tests executed only once (thus this
fwrd call)

> OK, I rechecked this, and BOOST_CHECK_CLOSE_FRACTION appears to have next to 
> no overhead now - excellent!

> >> * Exemplary error messages when things fail - Boost.Test has improved in
> >> this area, but IMO not enough.
> >
> > Specifically?
> 
> Testing:
> 
> double a = 1;
> double b = 2;
> BOOST_CHECK_CLOSE_FRACTION(a, b, 0.0);
> 
> Yields:
> 
> m:/data/boost/trunk/ide/libraries/scrap/scrap.cpp(41): error: in 
> "test_main_caller( argc, argv )": difference{0} between a{1} and b{2} 
> exceeds 0
> 
> Leaving aside the obvious bug (the difference is not zero!!), I would have 
> printed this as something like:
> 
> m:/data/boost/trunk/ide/libraries/scrap/scrap.cpp(41): error: in 
> "test_main_caller( argc, argv )": difference between a and b exceeds 
> specified tolerance with:
>    a                  =  1.0
>    b                  = 2.0
>    tolerance      = 0.0
>    difference     = 1.0
> 
> Which I'm sure will get mangled in email, but the idea is that the values 
> are pretty printed so they all line up nicely - makes it much easier to see 
> the problem compared to dumping them all on one line.

There are different opinions how output should look like. Some people prefer
multiline detailed description, some prefer single line consistent output. We
can probably provide an easier interface for output customization thus you can
use some simple plugins in your test modules to see it the way you like.
 
> >> * An easy way to tell if the last test has failed, and/or an easy way to
> >> print auxiliary information when the last test has failed.  This is
> >> primarily for testing in loops, when iterating over tabulated test data.
> >
> > This is addressed with trunk improvements. There is several tools 
> > introduced to
> > help with context specification.
> 
> Such as?  Docs?

BOOST_TEST_INFO
BOOST_TEST_CONTEXT

There is a unit test for these in test_tools_test.cpp. Used like these:

    BOOST_TEST_INFO( "info 1" );
    BOOST_TEST_INFO( "info 2" );
    BOOST_TEST_INFO( "info 3" );
    BOOST_CHECK( false );

    BOOST_TEST_CONTEXT( "some sticky context" ) {
        BOOST_CHECK( false );

        BOOST_TEST_INFO( "more context" );
        BOOST_CHECK( false );

        BOOST_TEST_INFO( "different subcontext" );
        BOOST_CHECK( false );
    }

context only reported if error occurred. Docs are pending.

 
> >> * Ultra stable code.  Exempting bug fixes, I'd like to see a testing 
> >> library
> >> almost never change, or only change after very careful consideration, for
> >> example if a new C++ language feature requires special testing support.
> >
> > Test library as any other library has users, bugs, feature requests etc. 
> > It has
> > a life on it's own. It does indeed need to be more carefully maintained in
> > comparison with other libs, but:
> >
> > 1. Proper component dependency helps. Your library needs to be built 
> > against
> > released version of Test library (even trunk one). This way Test library 
> > can do
> > it's own development in parallel.
> 
> The way Boost works at present is that if Trunk breaks, then so does my 
> stuff in Trunk: as a result I can no longer see whether I'm able to merge to 
> release or not if Boost.Test is broken on Trunk.  Unfortunately this has 
> happened to me a few times now.

Test library aside, are your expectations that either 

a) none of your dependencies ever change
b) any changes in every revision will work on every platform
c) your dependencies can only change when are not doing your own development
 
> > Again: proper component dependency. Depending on trunk version of your
> > dependencies is the root cause of the issue here. One library should 
> > depend on
> > specific released version of another library A.deps = B:1.2.3
> 
> And again that's not how Boost testing currently works, no matter what you 
> may wish.

Well in this case we need to be a bit more patient when dependent library is
changing.

> The issue here is that breaking changes should not be made to trunk without 
> checking to see what else in Boost is depending on those features.  As you 
> know that didn't happen with the last major commit.

I believe you overstate the issue. If there were any breakages these were
addressed within couple testing cycles.

Gennadiy