Re: [1.72][Test][Math][Fiber] Recent breaking change in Boost.Test breaks fiber/math tests

From: Raffi Enficiaud (raffi.enficiaud_at_[hidden])
Date: 2019-11-05 18:48:48

On 05.11.19 17:03, Paul A Bristow via Boost wrote:
>> [snip]
>>
>> Actually I like the change - did we need 'framework'?

"#me too" but this is not the right time to introduce any breaking
changes, especially without notice. It is used nowhere in boost.test but
I have not checked within boost (and certainly not outside) before
making that change (that was also unrelated to the issue being solved).
Some legacy code, a proper deprecation should be emitted, at least in
the change logs.

I do not know if namespaces can have deprecation attributes, that would
be handy.

>> Anyone using this is perhaps relying on a detail?
>>
>> (The need for it in the Math test case was to ensure that the
>> BOOST_TEST_MESSAGE("my_message") were actually visible.  I think it would be
>> better if messages were visible by default, not something one needs to ask for
>> specially.  I also was unconvinced the setting the environment variable
>> BOOST_TEST_MESSAGE worked but I'll check this out and raise a bug report.)
> 
> I've finally found this exchange explaining the behaviour.
> 
> http://boost.2283326.n4.nabble.com/test-Problem-with-BOOST-TEST-MESSAGE-td467539
> 8.html
> 
> I think it is very counter-intuitive that messages only emerge to the normal
> output if there is an error or --log_level="message"
> 
> And that 
> 
>    #define BOOST_TEST_LOG_LEVEL "message"
> 
>  does not appear to have the same effect as --log_level="message"

The current documentation never mentions the usage through a macro as
you are using in the snippet above. Please see below about the relevant
sections.

> 
> which is also surprising, and in my book a buglet?
> 
> I suggest that this behaviour should be changed so that BOOST_TEST_MESSAGE
> *always* produces output.
> 
> BOOST_AUTO_TEST_CASE(test_message)
> {
>    BOOST_TEST_MESSAGE("\nTest BOOST_TEST_MESSAGE.\n");
> } // BOOST_AUTO_TEST_CASE(test_message)
> 
> boost_test_message_test.vcxproj ->
> I:\Cpp\math\x64\Release\boost_test_message_test.exe
> 
> should output
> 
> Running 1 test case...
> 
> Test BOOST_TEST_MESSAGE.
> 
> *** No errors detected

I tend to disagree with that. What I am expecting from my tests is the
report of "true positives", ie. assertions that fails for good reasons
like regressions. All the rest I consider as information that distracts,
or information printed only for non-regular/normal use like debugging. I
would for instance put a message or a warning that I can easily activate
if something goes wrong and needs my attention, until I fix the issue.
Then I would go back to the normal mode.

This is also the approach taken by tools like ctest or b2: normal
logging level prints only a very high level status, verbose logs are
redirected to file for potential inspection, details are printed on error.

The behaviour of Boost.Test in that regard is documented in various
places, mostly in the runtime arguments (command line) and in the
logging part:

*
https://www.boost.org/doc/libs/1_71_0/libs/test/doc/html/boost_test/test_output/log_formats.html
*
https://www.boost.org/doc/libs/1_71_0/libs/test/doc/html/boost_test/test_output/log_formats/test_log_output.html
*
https://www.boost.org/doc/libs/1_71_0/libs/test/doc/html/boost_test/test_output/test_tools_support_for_logging/test_output_macro_message.html
(even with its own section)
*
https://www.boost.org/doc/libs/1_71_0/libs/test/doc/html/boost_test/utf_reference/rt_param_reference/log_level.html
(indicates the environment variable as well)
*
https://www.boost.org/doc/libs/1_71_0/libs/test/doc/html/boost_test/runtime_config.html
although the section about environment variables can be clarified

> 
> Otherwise I am sure I will not be the only person confused.
> 
> At the very least the documentation should highlight that ones expectation may
> not match what actually happens.
I have to say that the current doc is sometimes more than complete with
some redundant parts and confusing information. If you think of a better
way to organize it or a better wording ("me not English native"), please
let me know, I would be happy to improve it.

Thanks!
Raffi

PS: and sorry about breaking your code again so close to the beta.