[Boost-bugs] [Boost C++ Libraries] #11416: BOOST_AUTO_TEST_CASE_EXPECTED_FAILURES should result in a fail verdict if the expectation is not fullfilled

Subject: [Boost-bugs] [Boost C++ Libraries] #11416: BOOST_AUTO_TEST_CASE_EXPECTED_FAILURES should result in a fail verdict if the expectation is not fullfilled
From: Boost C++ Libraries (noreply_at_[hidden])
Date: 2015-06-24 14:23:18

#11416: BOOST_AUTO_TEST_CASE_EXPECTED_FAILURES should result in a fail verdict if
the expectation is not fullfilled
------------------------------+---------------------
 Reporter:  peters@…          |      Owner:  rogeeff
     Type:  Bugs              |     Status:  new
Milestone:  To Be Determined  |  Component:  test
  Version:  Boost 1.57.0      |   Severity:  Problem
 Keywords:                    |
------------------------------+---------------------
 Given this test:

 {{{
 BOOST_AUTO_TEST_CASE_EXPECTED_FAILURES(fail_test, 1)
 BOOST_AUTO_TEST_CASE( fail_test ) {
     BOOST_CHECK(1 == 1);
 }
 }}}

 I would assume that this test results in a overall failed test, but the
 macro only prints a message on the console: "Test case fail_test has fewer
 failures than expected" and does not touch the verdict of the test. This
 is IMHO useless, because its easy to miss (Especially if CI is used)

 I introduced that test case into our test suite to ensure that the tests
 are working correctly (before we had tests that did nothing and resulted
 in PASS, because NDEBUG was defined)

-- 
Ticket URL: <https://svn.boost.org/trac/boost/ticket/11416>
Boost C++ Libraries <http://www.boost.org/>
Boost provides free peer-reviewed portable C++ source libraries.