Re: [boost] Boost Test Library, expected failues, unexpected success

From: Noah Roberts (roberts.noah_at_[hidden])
Date: 2007-02-13 13:46:33

Dave Steffen wrote:
> Hi Folks,
> 
> We've been using the boost unit test library for over a year now, and
> we're very happy with it.  There's one little nit, though, and I'm
> curious about what people think it means, and what people have been
> doing about it.
> 
> The issue is "expected failures".
> 
> Expected failures are specified at the test case level, e.g.
> 
> BOOST_TEST_CASE( test_case_name, expected failures )
> {
> 	...
> }
> 
> 
> So, one could have a test case with, say, four assertions
> (a.k.a. BOOST_CHECK), and specify that you expect two to fail.  Fine.
> How do you know if the two that failed were the two you expected to
> fail?
> 
> One solution is "don't do that": have only one assertion per test
> case.  We find that to be extremely cumbersome; if it take 20 lines of
> code to set up an object and put it into a given state, we'd have to
> duplicate those 20 lines of code across multiple test cases (or,
> alternately, extract them into a helper function, which is annoying
> and not always possible).
> 
> I _think_ what I'd like to see is 'expected failures' marked, not at
> the test case level, but at the assertion level; that is, instead of
> specifying "N out of the following M assertions are expected to fail"
> (as is done now), specify "This assertion is expected to fail" on an
> assertion-by-assertion basis.
> 
> I can probably go hack this in to the library, but I'd prefer not to.
> Any other thoughts or solutions?

I'm curious why simply inverting the test is not enough.  Assuming a 
BOOST_CHECK_FAILS existed:

BOOST_CHECK_FAILS(someobject.doesntwork());

could be just as easily written as:

BOOST_CHECK(!sameobject.doesntwork());

It is provable, using basic boolean algebra, that there is no case when 
there is a check that couldn't be written with either method so I don't 
understand the requirement for adding the former.