[boost] Boost Test Library, expected failues, unexpected success

From: Dave Steffen (dgsteffen_at_[hidden])
Date: 2007-02-13 12:07:11

Hi Folks,

We've been using the boost unit test library for over a year now, and
we're very happy with it.  There's one little nit, though, and I'm
curious about what people think it means, and what people have been
doing about it.

The issue is "expected failures".

Expected failures are specified at the test case level, e.g.

BOOST_TEST_CASE( test_case_name, expected failures )
{
        ...
}

So, one could have a test case with, say, four assertions
(a.k.a. BOOST_CHECK), and specify that you expect two to fail.  Fine.
How do you know if the two that failed were the two you expected to
fail?

One solution is "don't do that": have only one assertion per test
case.  We find that to be extremely cumbersome; if it take 20 lines of
code to set up an object and put it into a given state, we'd have to
duplicate those 20 lines of code across multiple test cases (or,
alternately, extract them into a helper function, which is annoying
and not always possible).

I _think_ what I'd like to see is 'expected failures' marked, not at
the test case level, but at the assertion level; that is, instead of
specifying "N out of the following M assertions are expected to fail"
(as is done now), specify "This assertion is expected to fail" on an
assertion-by-assertion basis.

I can probably go hack this in to the library, but I'd prefer not to.
Any other thoughts or solutions?

Thanks!!

----------------------------------------------------------------------
Dave Steffen, Ph.D.                Disobey this command!
Software Engineer IV                 - Douglas Hofstadter
Numerica Corporation
dg_at_steffen a_at_t numerica d_at_ot us  (remove @'s to email me)