Re: [Boost-users] Boost.Test BOOST_TEST/BOOST_REQUIRE_MESSAGE message arg problems

Subject: Re: [Boost-users] Boost.Test BOOST_TEST/BOOST_REQUIRE_MESSAGE message arg problems
From: Raffi Enficiaud (raffi.enficiaud_at_[hidden])
Date: 2018-06-28 13:07:45

Le 20/05/2018 à 19:42, Olaf Peter via Boost-users a écrit :
> Hello Boost Users,
> 
> for my project I'm testing using BOOST_AUTO_TEST_SUITE and data driven 
> test sets (BOOST_DATA_TEST_CASE). The goal is to check the proper 
> working of a parser by checking success as such and compare the result 
> with a "expected" data (string) vector. In case of parse error, I want 
> to write the captured cout/cerr by boost.test into a file. In case of 
> compare error the parsed result also shall be written to a file. The 
> fragments show the concept/current state:
> 
> namespace x3_test { // aka utils ....
> 
> /* [Detect if boost test case failed](
>  Â * 
> https://stackoverflow.com/questions/18466857/detect-if-boost-test-case-failed?answertab=active#tab-top) 
> */
> bool current_test_passing()
> {
>  Â  using namespace boost::unit_test;
>  Â  auto const id = framework::current_test_case().p_id;
>  Â  auto const test_results = results_collector.results(id);
>  Â  return test_results.passed();
> }
> 
> std::string report_diagnostic(fs::path const& test_case_name, 
> std::string const& input, std::string const& result)
> {
>  Â Â Â  ...
>  Â Â Â  if(!x3_test::current_test_passing()) { Â Â Â  // only write in case of 
> failed test
>  Â Â Â Â Â Â Â  test_case_result_writer result_writer(test_case_name);
>  Â Â Â Â Â Â Â  result_writer.write(result);
>  Â Â Â  }
>  Â Â Â  return ss.str();
> }
> 
> } // x3_test
> 
> ...
> 
> BOOST_AUTO_TEST_SUITE(....
> 
> BOOST_DATA_TEST_CASE( declaration,
>  Â Â Â Â Â Â  dataset.input()
>  Â Â Â  ^ dataset.expect()
>  Â Â Â  ^ dataset.test_case_name(),
>  Â Â Â  code, expect_AST, test_case_name)
> {
>  Â Â Â  using attribute_type = ast::declaration;
>  Â Â Â  auto const parser = parser::declaration;
> 
>  Â Â Â  x3_test::testing_parser<attribute_type> parse;
>  Â Â Â  auto [parse_ok, parsed_AST] = parse(code, parser);
> 
>  Â Â Â  BOOST_TEST(parse_ok);
>  Â Â Â  BOOST_REQUIRE_MESSAGE(parse_ok,
>  Â Â Â Â Â Â Â  x3_test::report_diagnostic(test_case_name, code, parsed_AST)
>  Â Â Â  );
> 
>  Â Â Â  BOOST_TEST(parsed_AST == expect_AST, btt::per_element());
>  Â Â Â  BOOST_REQUIRE_MESSAGE(x3_test::current_test_passing(),
>  Â Â Â Â Â Â Â  x3_test::report_diagnostic(test_case_name, code, parsed_AST)
>  Â Â Â  );
> }
> 
> BOOST_AUTO_TEST_SUITE_END()
> 
> After some testing, the error count is twice the real (obviously here) 
> but it's all above a work arround.
> 
> It seems, that BOOST_REQUIRE_MESSAGE evaluates always the message 
> predicate (shown by using --log_level=all) nevertheless the test case 
> successed or failed. If failed it is streamed to the console.

That sounds odd to me. Would it be possible to check if the messages get 
evaluated in case you do not pass --log_level=all ?

Otherwise, if you be kind of you if you file a bug.

> 
> My first thoughs where that the message is called only in case of test 
> failure which isn't true.
> 
> The problem is now, that my report_diagnostic function write the test 
> output (parsed_AST) to a special dir/file for later evaluating etc. The 
> intuitive way would be:
> 
> std::string report_diagnostic(fs::path const& test_case_name, 
> std::string const& input, std::string const& result)
> {
>  Â Â Â  ...
>  Â Â Â  test_case_result_writer result_writer(test_case_name);
>  Â Â Â  result_writer.write(result);
> 
>  Â Â Â  return ss.str();
> }
> 
> BOOST_REQUIRE_MESSAGE(parse_ok,
>  Â Â Â Â Â Â Â  // call diagnostic only in failure case and write then
>  Â Â Â Â Â Â Â  x3_test::report_diagnostic(test_case_name, code, parsed_AST)
>  Â Â Â  );
> BOOST_REQUIRE_MESSAGE((parsed_AST == expect_AST, btt::per_element()),
>  Â Â Â Â Â Â Â  // call diagnostic only in failure case and write then
>  Â Â Â Â Â Â Â  x3_test::report_diagnostic(test_case_name, code, parsed_AST)
>  Â Â Â  );
> 
> Hence not depending on current_test_passing() any more.
> 
> How to get this?

What about :

1- using test-case fixtures for writing things at the end of the test
2- registering an observer for getting the failure signal?

The observer is (kind of) described here:
https://www.boost.org/doc/libs/master/libs/test/doc/html/boost/unit_test/framework/register_observer.html

You may install and remove it globally by using a global fixture:

https://www.boost.org/doc/libs/master/libs/test/doc/html/boost_test/utf_reference/test_org_reference/test_org_boost_test_global_fixture.html

Once registered, it will capture the events that you want, especially 
the failure cases.

 From the observer, you can get the current test cases with this

https://www.boost.org/doc/libs/master/libs/test/doc/html/boost/unit_test/framework/current_test_case.html

> 
> 
> Further, how can I get the captured std::cerr in case of '!parse_ok' 
> inside report_diagnostic?

Currently you have to do it manually. See this
https://stackoverflow.com/a/5419388/1617295

This can again go to a fixture of your own. There is a feature request 
on the backlog though:

https://svn.boost.org/trac10/ticket/12452

Best,
Raffi