Re: [Boost-users] Boost.Test BOOST_TEST/BOOST_REQUIRE_MESSAGE message arg problems

Subject: Re: [Boost-users] Boost.Test BOOST_TEST/BOOST_REQUIRE_MESSAGE message arg problems
From: Olaf Peter (ope-devel_at_[hidden])
Date: 2018-07-17 19:33:25

Hello Raffi,
>
>> BOOST_AUTO_TEST_SUITE(....
>>
>> BOOST_DATA_TEST_CASE( declaration,
>> Â Â Â Â Â Â Â  dataset.input()
>> Â Â Â Â  ^ dataset.expect()
>> Â Â Â Â  ^ dataset.test_case_name(),
>> Â Â Â Â  code, expect_AST, test_case_name)
>> {
>> Â Â Â Â  using attribute_type = ast::declaration;
>> Â Â Â Â  auto const parser = parser::declaration;
>>
>> Â Â Â Â  x3_test::testing_parser<attribute_type> parse;
>> Â Â Â Â  auto [parse_ok, parsed_AST] = parse(code, parser);
>>
>> Â Â Â Â  BOOST_TEST(parse_ok);
>> Â Â Â Â  BOOST_REQUIRE_MESSAGE(parse_ok,
>> Â Â Â Â Â Â Â Â  x3_test::report_diagnostic(test_case_name, code, parsed_AST)
>> Â Â Â Â  );
>>
>> Â Â Â Â  BOOST_TEST(parsed_AST == expect_AST, btt::per_element());
>> Â Â Â Â  BOOST_REQUIRE_MESSAGE(x3_test::current_test_passing(),
>> Â Â Â Â Â Â Â Â  x3_test::report_diagnostic(test_case_name, code, parsed_AST)
>> Â Â Â Â  );
>> }
>>
>> BOOST_AUTO_TEST_SUITE_END()
>>
>> After some testing, the error count is twice the real (obviously 
>> here) but it's all above a work arround.
>>
>> It seems, that BOOST_REQUIRE_MESSAGE evaluates always the message 
>> predicate (shown by using --log_level=all) nevertheless the test case 
>> successed or failed. If failed it is streamed to the console.
>
> That sounds odd to me. Would it be possible to check if the messages 
> get evaluated in case you do not pass --log_level=all ?
I reduced it and found the original problem. Checking parse_ok twice 
results into two messages, this explain what I faced (and misinterpreted 
in the last mail). The reason for introducing this was the intention of:

- check on parse_ok success
- if not, generate a diagnostic message - all further checks can be canceled
- compare the result per_element with reference/gold data
- if failed, generate a diagnostic message
- note, report_diagnostic saves the parsed_AST for later inspection

The problem may rise due to my limited understanding about the macros 
arround.

>
> What about :
>
> 1- using test-case fixtures for writing things at the end of the test
> 2- registering an observer for getting the failure signal?
>
> The observer is (kind of) described here:
> https://www.boost.org/doc/libs/master/libs/test/doc/html/boost/unit_test/framework/register_observer.html 
>
>
> You may install and remove it globally by using a global fixture:
>
> https://www.boost.org/doc/libs/master/libs/test/doc/html/boost_test/utf_reference/test_org_reference/test_org_boost_test_global_fixture.html 
>
>
> Once registered, it will capture the events that you want, especially 
> the failure cases.
>
> From the observer, you can get the current test cases with this
>
> https://www.boost.org/doc/libs/master/libs/test/doc/html/boost/unit_test/framework/current_test_case.html 
>
I will check/evaluate it next time, it simplifies diagnostic since the 
parsed_AST can be written by the observer

Thank you,
Olaf