Re: [Boost-users] [Boost.Test] BOOST_FAIL vs BOOST_ERROR

Subject: Re: [Boost-users] [Boost.Test] BOOST_FAIL vs BOOST_ERROR
From: Gennadiy Rozental (rogeeff_at_[hidden])
Date: 2009-05-12 23:53:34

Greg Christopher wrote:
>>>   I think that _both_ BOOST_FAIL (or BOOST_REQUIRE) and BOOST_ERROR
>>> (or BOOST_CHECK) do indicate that there is a software under test
>>> failure (bug
>> They can indicate whatever you want it to indicate. If you require some
>> particular condition before proceeding with test you check for it using
>> BOOST_REQUIRE on top of the test case. You can also use fixtures.
> 
> Right, but the point is there are two reasons why I might not be able to continue:
> 1)A feature of the software under test is broken, and therefore I won't be able to test further. Still a bug in the software I'm testing.
> 2)Something exceptional has happened outside the ability of my software under test to deal with. At this point, I want to abend without indicating a software "bug".
> 
> I do think fixtures are a great way to go. If the fixture initialize fails, don't run that test and report a problem in the testware or environment. But sometimes I may want to indicate that during the flow of the testing. 

I believe, at the moment fixture execution is part of the test case and 
is treated accordingly.

> I think that test reporting frameworks need at least three states they indicate- fail(bug), success, or "other". At a higher level, I want the system to _only_ contact the team executing the tests if "other" comes up so they can look at the testing system. As such systems over-report "failures" or bugs, trust in the testing system erodes.

Boost.Test tools already has 3 levels: CHECK, REQUIRE and WARN. You can 
use third level to report conditions which are not software bugs. You'll 
need to abort the test case manually though, since Boost.Test continues 
on warning. The warnings are reported accordingly in an output.

Gennadiy