Re: [Boost-users] [Boost.Test] reporting tests with warnings

Subject: Re: [Boost-users] [Boost.Test] reporting tests with warnings
From: Richard (legalize+jeeves_at_[hidden])
Date: 2011-11-18 19:27:55

[Please do not mail me a copy of your followup]

boost-users_at_[hidden] spake the secret code
<loom.20111117T062902-477_at_[hidden]> thusly:

>Richard <legalize+jeeves <at> mail.xmission.com> writes:
>> >Boost.Test already support three "symbols":
>> >
>> >  BOOST_REQUIRE_*
>> >  BOOST_CHECK_*
>> >  BOOST_WARN_*
>> >
>> >What you're seems to be implying is that BOOST_WARN_* is a bad idea.
>> 
>> Not just BOOST_WARN_*, but also BOOST_CHECK_* since they keep
>> executing a failed test case.
>
>Why so radical view?

I don't think its radical to want my tests to simply stop at the
first point of failure.

>This is TDD purist position and is not necessarily the only
>one. 

I don't know if its "purist" or not and that's beside the point.

I never said it was the "only" position.

>> If you keep executing a test case
>> that's already failed, it is highly likely in C++ that the test
>> runner will simply crash.
>
>This is simply not true.

So if I do something like:

BOOST_CHECK(p);
BOOST_CHECK(p->accessor());

and p is a null pointer, what good does it do to continue after the
first failed check?  The pointer is null and anything else that is
checked after that is useless.  Dereferencing a null pointer is going
to crash.  Yes, on Windows that crash is turned into an exception
that the test runner could intercept, but its still worthless.  We
don't even know if the whole heap has been corrupted at this point.

The bottom line is that due to the low-level nature of C++, all kinds
of things could go wrong in the test runner's environment that
are outside the control of the test runner.  Yes, you can guard
against most kinds of controlled failures, but you simply can't
guarantee that C++ test code (or the SUT) didn't horribly screw
everything up.  In managed languages like C# and Java, this simply
isn't the case due to their different execution model.

That's why for C++ code, I consider it a bad idea to use anything
other than BOOS_REQUIRE.  I want the test case to fail immediately
and not continue any further at all once an expectation hasn't been
met.

Yes, its my opinion, but its not one that I grabbed onto randomly.

Even in other environments (C#, for instance), I don't have any use
for warnings or "mark a failure, but keep executing the test case".
Whenever I've encountered this in other people's tests, its because
the test is trying to test too many things instead of a single thing
and so they are trying to report multiple different possible failures
from a single test case.  A single test should test one thing only.
You may call this "TDD purism", but I simply call it good test
design and it seems to be an opinion shared among other people I know
that have been writing unit tests for a long time and have much more
experience at it than I do.
-- 
"The Direct3D Graphics Pipeline" -- DirectX 9 version available for download
 <http://legalizeadulthood.wordpress.com/the-direct3d-graphics-pipeline/>
      Legalize Adulthood! <http://legalizeadulthood.wordpress.com>