Re: [boost] What Should we do About Boost.Test?

Subject: Re: [boost] What Should we do About Boost.Test?
From: Dave Abrahams (dave_at_[hidden])
Date: 2012-10-04 13:20:00

Sorry I went quiet for a while; I got sick.  I figure I should respond
directly to this message, even if I don't respond to any others...

on Mon Sep 17 2012, Gennadiy Rozental <rogeeff-AT-gmail.com> wrote:

> Dave Abrahams <dave <at> boostpro.com> writes:
>
>> Hi All, 
>> 
>> I was just going through Boost.Test to try to figure out how to teach
>> it, and while it looks to have substantial value, it is also in quite
>> a mess.
>
>> It contains loads of features that are exercised in the examples/
>> directory but neither included in any of the tests nor documented.
>
> There are not that many of these in fact. They can be split into 2 kinds:
>
> * Implemented long time ago and never documented
> There couple of these, related to interaction based testing and mock object 
> support
> * Brand new features
> There are series of new features implemented about a year ago. Almost ready for
> prime time, but not 100%. Specifically lacking documentation.

The question remains: "how do I learn/teach this library?"  If I can't
answer those questions, I also can't answer the question

       How do I use this library?

I don't understand how other people have arrived at answers for
themselves.

>> There are facilities for command-line argument parsing!
>
> Yes, indeed and while I like these much better than official boost
> one, I do not insist it to be a public interface, thus it does not
> need to be documented.

That's fine, but in the presence of so many other problems and of a
large suite of examples/ directed at this feature, it contributes to the
sense of uncertainty about what this library *is*.  BTW, also, it's
completely unclear how CLA processing relates to the mission of the
library.
 
>> There are
>> "decorators" that turn on/off features for test cases.
>
> This is new feature. 
>
>> There is support for mock objects! 
>> These are cool and sometimes necessary features, but who knew?
>
>> The third tutorial page
>> has a glaring typo in the code examples: "BOOST_AUTO_EST_CASE".
>> There's no reference manual at all.
>
> 1. There is a significantly improved documentation in the trunk. I never got to
> release it, just to avoid rocking the boat (and I hoped just to do one big
> release with all new improvements)

Does that "improved documentation" apply to what's on the release
branch, or only to what's available in trunk?

> 2. There is no *formal* reference documentation, but I am not convinced there 
> is a huge need for one. 

There most certainly is, *especially* in the presence of so much other
uncertainty.

> In majority of the cases Boost.Test unlike other boost libraries is
> not extended or accessed through class interface.

I don't see how that's relevant.

> There are few interfaces which are indeed used and they are
> documented.

??? I can't even begin to understand how you can say that.  Everything
one does with a library, one does through an interface.  Every interface
needs to be documented so that users know how to use it correctly.
Otherwise, it's just your private code.

>> There are nearly-identical files in the examples/ directory called
>> "est_example1.cpp" and "test_example1.cpp" (Did the "t" key on
>> someone's keyboard break?)  I could go on, but where would I stop?
>
> These are two completely different things: est is "exception safety test".
> Should have named it different I guess ;)

I guess.  Is BOOST_AUTO_EST_CASE another example of "exception safety
test," or is that a genuine typo?

>> I don't know what to do about this.  Because of the lack of redundancy
>> (i.e. tests and documentation), it's hard to tell whether this library
>> is correct or even to define what "correct" should mean.  It seems like,
>
> I am not sure what you mean. There is an extensive self test modules.

I mean *redundancy* between the tests and the documentation.  The tests
should check that the library does what the documentation says it does.
>From the tests alone I can't even draw a conclusion about what you
intend as a stable, supported, public interface.

>> I am not at all attached to removing Boost.Test from Boost, but IMO
>> rescuing it would require a significant new investment of time and
>> energy from people who are committed to bringing the library up to par
>> with the rest of what we do.
>
> I an not quite convinced that anything is really in such a bad shape
> that it requires rescuing. That said if anyone is interested in
> helping to bring up latest release I am happy to share the load.

Look, I teach classes on Boost.  If Boost.Test is not learnable and
teachable, I have to tell my students to stay away from it.  That's
embarrassing for me, and bad for Boost.

-- 
Dave Abrahams
BoostPro Computing                  Software Development        Training
http://www.boostpro.com             Clang/LLVM/EDG Compilers  C++  Boost