Re: [boost] doctest - the lightest feature rich C++ single header testing framework - if it can enter boost and if it/boost will benefit from that

Subject: Re: [boost] doctest - the lightest feature rich C++ single header testing framework - if it can enter boost and if it/boost will benefit from that
From: Gavin Lambert (gavinl_at_[hidden])
Date: 2016-05-22 23:26:43

On 23/05/2016 14:29, Vinnie Falco wrote:
> I'm reading this thread about doctest and I checked out the library.
> The first thing I noticed was macros and odd syntax. For example:
>
>      TEST_CASE("testing the factorial function") {
>         CHECK(factorial(1) == 1);
>         CHECK(factorial(2) == 2);
>         CHECK(factorial(10) == 3628800);
>      }
>
> This looks like non-standard C++ and heavy on macros which turned me
> off to Boost.Test for the same reasons. I considered using Boost.Test
> for yet another library I am about to release and I gave up when I saw
> the baffling use of macros.

Generally macros are used for assertions because you want to include 
__LINE__ (and typically also __FILE__) in the assertion failure so that 
you can find *which* assertion failed (often the condition alone is not 
sufficiently unambiguous), and there isn't really any better way to get 
those AFAIK. (Sometimes it's possible to extract the failure location 
from a thrown exception, but this is highly compiler-dependent and 
non-standard -- and you're not interested in that specific point, but in 
one of its callers.)

Using macros for the test case definition is more avoidable but is still 
generally done to avoid boilerplate (in this case, presumably at least 
auto-registration).

> People seem to be lavishing praise on these test libraries so I am
> asking sincerely, what am I missing here? I used my own unit test
> framework which is header-only, customizable, doesn't require macros
> in the tests, and has a test structure which is easy to generate XML
> reports from. Here it is:
> https://github.com/vinniefalco/Beast/blob/master/extras/beast/unit_test/suite.hpp

Your version seems to not capture line numbers and thus requires coming 
up with a unique string reason for each assertion, which is cumbersome.

> To write a test you subclass from `beast::unit_test::suite` and
> implement the `run()` override, and then call `expect()` for each
> condition that should be true. I was able to get Beast to almost 97%
> coverage using this simple system (see
> https://codecov.io/gh/vinniefalco/Beast). Here's an example of a test:
> https://github.com/vinniefalco/Beast/blob/master/test/core/basic_streambuf.cpp#L170

This also requires listing each test twice (once in implementation, once 
in run()), and each suite twice (one in implementation, once in 
BEAST_DEFINE_TESTSUITE... which is one of those macros you didn't like, 
isn't it?).

Requiring separate listing is "old school" test runner design; it went 
out of fashion because it's too error-prone (it's easy to forget to list 
a test case, and if you're not ensuring that tests fail before you make 
them pass then you might think the test passed instead of simply not 
being run).

This design also makes it harder to run or debug individual test cases 
within a suite.  (I don't know how easy doctest makes this, but most of 
the popular test runners can do this very easily, without recompiling.) 
  You can comment out some of them but this requires a rebuild and is 
also error-prone since you have to remember to uncomment them again later.

If it works for you, that's fine -- but these are some of the reasons 
why test frameworks generally avoid those things and use macros and 
auto-registration instead.