Re: [1.72][Test][Math][Fiber] Recent breaking change in Boost.Test breaks fiber/math tests

From: pbristow_at_[hidden]
Date: 2019-11-06 09:49:46

There are times when eminently sensible suggestions (like yours and mine) are
not the best thing to do.

I agree that the benefit isn't worth causing downstream trouble.

But I would still like some warning in the documentation.  I've written
notes-to-self:

// #define BOOST_TEST_LOG_LEVEL "all"   or /DBOOST_TEST_LOG_LEVEL // Do not
output unless there is an error!
//
http://boost.2283326.n4.nabble.com/test-Problem-with-BOOST-TEST-MESSAGE-td467539
8.html
// run with --log_level="message" always gives message.

I use a VS IDE Custom build step "$(TargetDir)$(TargetName).exe"
--log_level="message" for testing console apps.

Paul

> -----Original Message-----
> From: Boost <boost-bounces_at_[hidden]> On Behalf Of Raffi Enficiaud via
> Boost
> Sent: 5 November 2019 18:49
> To: boost_at_[hidden]
> Cc: Raffi Enficiaud <raffi.enficiaud_at_[hidden]>
> Subject: Re: [boost] [1.72][Test][Math][Fiber] Recent breaking change in
Boost.Test
> breaks fiber/math tests
> 
> On 05.11.19 17:03, Paul A Bristow via Boost wrote:
> >> [snip]
> >>
> >> Actually I like the change - did we need 'framework'?
> 
> "#me too" but this is not the right time to introduce any breaking changes,
> especially without notice. It is used nowhere in boost.test but I have not
checked
> within boost (and certainly not outside) before making that change (that was
also
> unrelated to the issue being solved).
> Some legacy code, a proper deprecation should be emitted, at least in the
change
> logs.
> 
> I do not know if namespaces can have deprecation attributes, that would be
> handy.
> 
> >> Anyone using this is perhaps relying on a detail?
> >>
> >> (The need for it in the Math test case was to ensure that the
> >> BOOST_TEST_MESSAGE("my_message") were actually visible.  I think it
> >> would be better if messages were visible by default, not something
> >> one needs to ask for specially.  I also was unconvinced the setting
> >> the environment variable BOOST_TEST_MESSAGE worked but I'll check
> >> this out and raise a bug report.)
> >
> > I've finally found this exchange explaining the behaviour.
> >
> > http://boost.2283326.n4.nabble.com/test-Problem-with-BOOST-TEST-MESSAG
> > E-td467539
> > 8.html
> >
> > I think it is very counter-intuitive that messages only emerge to the
> > normal output if there is an error or --log_level="message"
> >
> > And that
> >
> >    #define BOOST_TEST_LOG_LEVEL "message"
> >
> >  does not appear to have the same effect as --log_level="message"
> 
> The current documentation never mentions the usage through a macro as you are
> using in the snippet above. Please see below about the relevant sections.
> 
> >
> > which is also surprising, and in my book a buglet?
> >
> > I suggest that this behaviour should be changed so that
> > BOOST_TEST_MESSAGE
> > *always* produces output.
> >
> > BOOST_AUTO_TEST_CASE(test_message)
> > {
> >    BOOST_TEST_MESSAGE("\nTest BOOST_TEST_MESSAGE.\n"); } //
> > BOOST_AUTO_TEST_CASE(test_message)
> >
> > boost_test_message_test.vcxproj ->
> > I:\Cpp\math\x64\Release\boost_test_message_test.exe
> >
> > should output
> >
> > Running 1 test case...
> >
> > Test BOOST_TEST_MESSAGE.
> >
> > *** No errors detected
> 
> I tend to disagree with that. What I am expecting from my tests is the report
of
> "true positives", ie. assertions that fails for good reasons like regressions.
All the
> rest I consider as information that distracts, or information printed only for
non-
> regular/normal use like debugging. I would for instance put a message or a
warning
> that I can easily activate if something goes wrong and needs my attention,
until I
> fix the issue.
> Then I would go back to the normal mode.
> 
> This is also the approach taken by tools like ctest or b2: normal logging
level prints
> only a very high level status, verbose logs are redirected to file for
potential
> inspection, details are printed on error.
> 
> The behaviour of Boost.Test in that regard is documented in various places,
mostly
> in the runtime arguments (command line) and in the logging part:
> 
> *
> https://www.boost.org/doc/libs/1_71_0/libs/test/doc/html/boost_test/test_outp
> ut/log_formats.html
> *
> https://www.boost.org/doc/libs/1_71_0/libs/test/doc/html/boost_test/test_outp
> ut/log_formats/test_log_output.html
> *
> https://www.boost.org/doc/libs/1_71_0/libs/test/doc/html/boost_test/test_outp
> ut/test_tools_support_for_logging/test_output_macro_message.html
> (even with its own section)
> *
> https://www.boost.org/doc/libs/1_71_0/libs/test/doc/html/boost_test/utf_refere
> nce/rt_param_reference/log_level.html
> (indicates the environment variable as well)
> *
> https://www.boost.org/doc/libs/1_71_0/libs/test/doc/html/boost_test/runtime_c
> onfig.html
> although the section about environment variables can be clarified
> 
> 
> 
> >
> > Otherwise I am sure I will not be the only person confused.
> >
> > At the very least the documentation should highlight that ones
> > expectation may not match what actually happens.
> I have to say that the current doc is sometimes more than complete with some
> redundant parts and confusing information. If you think of a better way to
> organize it or a better wording ("me not English native"), please let me know,
I
> would be happy to improve it.
> 
> Thanks!
> Raffi
> 
> PS: and sorry about breaking your code again so close to the beta.
> 
> 
> _______________________________________________
> Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost