Re: [Boost-users] [Boost.Test] BOOST_FAIL vs BOOST_ERROR

Subject: Re: [Boost-users] [Boost.Test] BOOST_FAIL vs BOOST_ERROR
From: Greg Christopher (gchristopher_at_[hidden])
Date: 2009-05-08 12:47:44

Hi Gennadiy ; thanks very much for your response...

> -----Original Message-----
> From: boost-users-bounces_at_[hidden] [mailto:boost-users-
> bounces_at_[hidden]] On Behalf Of Gennadiy Rozental
> Sent: Friday, May 08, 2009 1:04 AM
> To: boost-users_at_[hidden]
> Subject: Re: [Boost-users] [Boost.Test] BOOST_FAIL vs BOOST_ERROR
> 
> Greg Christopher <gchristopher <at> vmware.com> writes:
> 
> > For instance, a reporting entry point that says "this is an
> > exceptional condition and not the detection of a software bug".
> 
> Condition of what then?

For example, network down.

> 
> >   I think that _both_ BOOST_FAIL (or BOOST_REQUIRE) and BOOST_ERROR
> > (or BOOST_CHECK) do indicate that there is a software under test
> > failure (bug
> 
> They can indicate whatever you want it to indicate. If you require some
> particular condition before proceeding with test you check for it using
> BOOST_REQUIRE on top of the test case. You can also use fixtures.

Right, but the point is there are two reasons why I might not be able to continue:
1)A feature of the software under test is broken, and therefore I won't be able to test further. Still a bug in the software I'm testing.
2)Something exceptional has happened outside the ability of my software under test to deal with. At this point, I want to abend without indicating a software "bug".

I do think fixtures are a great way to go. If the fixture initialize fails, don't run that test and report a problem in the testware or environment. But sometimes I may want to indicate that during the flow of the testing. 

I think that test reporting frameworks need at least three states they indicate- fail(bug), success, or "other". At a higher level, I want the system to _only_ contact the team executing the tests if "other" comes up so they can look at the testing system. As such systems over-report "failures" or bugs, trust in the testing system erodes.

I also feel strongly that this has to be done within the reporting APIs, and can't be done via string conventions. Eventually conventions fail and systems that parse the logs will improperly report bugs. APIs for reporting should handle formatting and indirectly, control flow at the triage level where people respond to problems. I think everyone is in agreement on that- I just wonder if you handle this third state as far as reporting.

Thanks!

Greg

> 
> 
> > properly detected by testware). I'd like to hear I'm wrong or that
> > there is another way to signal a testware failure.
> >
> >   I suppose that I can use fixture support to attempt to notice any
> > exceptional problems during setup so that I don't even get to the
> > test execution code when there is an exceptional condition or
> > environment problem. However, I'd like to try to wrap my head around
> > the capabilities of the system and understand them as best as I can
> > before proceeding.
> 
> Please let me know if understood your problem.
> 
> Gennadiy
> 
> _______________________________________________
> Boost-users mailing list
> Boost-users_at_[hidden]
> http://lists.boost.org/mailman/listinfo.cgi/boost-users