Re: [boost] Boost Test Library, expected failues, unexpected success

From: Dave Steffen (dgsteffen_at_[hidden])
Date: 2007-02-13 13:57:59

Noah Roberts writes:
 > Dave Steffen wrote:
 > > Hi Folks,
[...]
 > > Expected failures are specified at the test case level, e.g.
 > > 
 > > BOOST_TEST_CASE( test_case_name, expected failures )
 > > {
 > > 	...
 > > }
[...]
 > > I _think_ what I'd like to see is 'expected failures' marked, not at
 > > the test case level, but at the assertion level; that is, instead of
 > > specifying "N out of the following M assertions are expected to fail"
 > > (as is done now), specify "This assertion is expected to fail" on an
 > > assertion-by-assertion basis.
 > > 
 > > I can probably go hack this in to the library, but I'd prefer not to.
 > > Any other thoughts or solutions?
 > 
 > I'm curious why simply inverting the test is not enough.  Assuming
 > a BOOST_CHECK_FAILS existed:
 > 
 > BOOST_CHECK_FAILS(someobject.doesntwork());
 > 
 > could be just as easily written as:
 > 
 > BOOST_CHECK(!sameobject.doesntwork());

 Well, that looks like any other test assertion, and results in a pass
 or fail.  What we're after here is something different: we're
 distinguishing between two different kinds of failures, and we want
 them reported as such.  In contrast, what you've got above turns an
 "expected failure" into a "pass", which isn't what we want.

 There is a larger question, that is probably better directed at
 Gennady, since he wrote the library: why support expected failures?
 What do "expected failures" mean?  What's the use case?  This is,
 however, a different discussion (that we can have if people are
 interested).

 But the point is that failures and expected failures are
 reported differently.  There are, in effect, three possible outcomes
 of a test: pass, fail, and fail (but we expected it to).

 What I'm asking for is that the "expected failure" notion be
 specified, not at the test case level, but at the test assertion
 level... but with the same reporting scheme as is currently in use.

----------------------------------------------------------------------
Dave Steffen, Ph.D.      
Software Engineer IV              Disobey this command!
Numerica Corporation
ph [phone] x227    
fax [phone]                   - Douglas Hofstadter
dgsteffen_at_[hidden]
___________________
Numerica Disclaimer:
This message and any attachments are intended only for the individual
or entity to which the message is addressed.  It is proprietary and
may contain privileged information.  If you are neither the intended
recipient nor the agent responsible for delivering the message to the
intended recipient, you are hereby notified that any review,
retransmission, dissemination, or taking of any action in reliance
upon, the information in this communication is strictly prohibited,
and may be unlawful.  If you feel you have received this communication
in error, please notify us immediately by returning this Email to the
sender and deleting it from your computer.