[Boost-bugs] [Boost C++ Libraries] #3444: BOOST_CHECK_MESSAGE - 2 problems

Subject: [Boost-bugs] [Boost C++ Libraries] #3444: BOOST_CHECK_MESSAGE - 2 problems
From: Boost C++ Libraries (noreply_at_[hidden])
Date: 2009-09-14 04:01:24

#3444: BOOST_CHECK_MESSAGE - 2 problems
------------------------------------------------------+---------------------
 Reporter:  Maxim Yanchenko <Maxim.Yanchenko@…>       |       Owner:  rogeeff
     Type:  Bugs                                      |      Status:  new    
Milestone:  Boost 1.41.0                              |   Component:  test   
  Version:  Boost 1.38.0                              |    Severity:  Problem
 Keywords:                                            |  
------------------------------------------------------+---------------------
 1) The 2nd argument is always evaluated.
 I believe it's natural for a user to expect that BOOST_CHECK_MESSAGE
 follows short circuit logic, i.e. if the check was satisfied, no more user
 code is called.

 Example:
 {{{
 BOOST_CHECK_MESSAGE( x == y, report_non_equal(x,y) )
 }}}

 report_non_equal(x,y) should be evaluated only if the check failed, as a
 natural precondition for the report_non_equal is that its arguments are
 not equal (e.g. a user can divide something by the difference of x and y
 without checking that it's not zero).

 2) The message is printed even in case of success with
 --log_level=success.
 People usually place there a message like "failed" in the message.
 For example, see the Boost regression test suite (just picked a couple of
 examples):
 {{{
 ./libs/test/test/test_tools_test.cpp:210:    BOOST_CHECK_MESSAGE(
 test_pred1(), "Checking predicate failed" )
 ./libs/iostreams/test/large_file_test.cpp:382:    BOOST_CHECK_MESSAGE(amt
 == 1, "failed reading character");
 ./libs/serialization/test/test_non_intrusive.cpp:152:
 BOOST_CHECK_MESSAGE(pa1 == &a, "Copy of pointer not correctly restored");
 }}}
 And the Boost.Test docs is not an exception:
 {{{
 BOOST_WARN_MESSAGE( res > 1, "sin(45){" << res << "} is <= 1. Hmm..
 Strange. " );
 }}}
 As you can see, the message users provide usually describes the failure
 ("X failed"), not the test itself ("testing X").
 So when they use --log_level=success they are very confused when they see
 that something successfully failed (as a message from UTF "check 'X
 failed' passed" means that the failure X was expected).

 In my case, I used the message to report the position where two strings
 mismatched using simple "std::mismatch-begin" (naturally expecting it to
 be called only in a case of failure), so with --log_level=success it
 printed that all strings mismatched in their end with funny and ambiguous
 messages like
 "info: check 'Strings "asd" and "asd" mismatch at position 3' passed"
 (does it mean that they were expected to mismatch at pos 3?), while they
 worked perfectly when the check failed:
 "error in "test": Strings "asd" and "asx" mismatch at position 2"

 I believe if you run Boost regression test suite with --log_level=success
 you will see a lot of similar absurd messages.



 From the first glance, the fix for both problems seems obvious (haven't
 tried yet, though) - in case of BOOST_<level>_MESSAGE the following
 version of BOOST_CHECK_IMPL should be used:
 {{{
 #define BOOST_CHECK_IMPL( P, check_descr, TL, CT )
 \
 do {
 \
     BOOST_TEST_PASSPOINT();
 \
     const bool result = P;
 \
     if (result)
 \
       BOOST_TEST_TOOL_IMPL( check_impl, result, BOOST_TEST_STRINGIZE( P ),
 TL, CT ), 0 );         \
     else
 \
       BOOST_TEST_TOOL_IMPL( check_impl, result, check_descr, TL, CT ), 0
 );\
 } while( ::boost::test_tools::dummy_cond )
 \
 /**/

 }}}

-- 
Ticket URL: <https://svn.boost.org/trac/boost/ticket/3444>
Boost C++ Libraries <http://www.boost.org/>
Boost provides free peer-reviewed portable C++ source libraries.