[Boost-bugs] [Boost C++ Libraries] #6129: Include warning counts in reports

Subject: [Boost-bugs] [Boost C++ Libraries] #6129: Include warning counts in reports
From: Boost C++ Libraries (noreply_at_[hidden])
Date: 2011-11-17 08:37:51

#6129: Include warning counts in reports
---------------------------------------+------------------------------------
 Reporter:  olaf.meeuwissen@…          |       Owner:  rogeeff
     Type:  Feature Requests           |      Status:  new    
Milestone:  To Be Determined           |   Component:  test   
  Version:  Boost 1.47.0               |    Severity:  Problem
 Keywords:                             |  
---------------------------------------+------------------------------------
 I [http://lists.boost.org/boost-users/2011/11/71744.php mentioned] this on
 the mailing list before, but I would like to be able to report the number
 of times assertions in the `BOOST_WARN` "family" have failed.

 For test cases, the (XML) report includes information similar to:
 {{{
 result="passed"
 assertions_passed="x" assertions_failed="x" expected_failures="x"
 }}}
 and for test suites
 {{{
 result="passed"
 assertions_passed="x" assertions_failed="x" expected_failures="x"
 test_cases_passed="x" test_cases_failed="x"
 test_cases_skipped="x" test_cases_aborted="x"
 }}}

 I guess I would like to see a count of something like `assertions_ignored`
 added to the test cases and an additional `ignored` value for their
 `result` attribute.  This `ignored` value is used for test cases that fail
 one or more of their `BOOST_WARN` type assertions but pass ''all'' of
 their `BOOST_REQUIRE` and `BOOST_CHECK` assertions.  If all assertions
 pass, the test case passes.  If a `BOOST_CHECK` assertion fails the test
 case fails.  If a `BOOST_REQUIRE` assertion fails, the test case is
 aborted and fails.

 For the test suites, the same story goes for the `assertions_ignored` and
 `ignored` additions, where an `ignored` result is assigned if one or more
 test cases with that result were encountered and none of the test cases
 `failed`.  If all test cases passed, the test suite passes.  If one or
 more test cases failed, the test suite fails.  In addition to the existing
 `test_cases_*` counts, I'd like to see a `test_cases_ignored` count.

 Summarizing, you'd get something like this:

  - counts of assertions `passed`, `ignored` or `failed`
  - a value of `passed`, `ignored` or `failed` for the `result` attribute
  - counts of test cases `passed`, `ignored`, `failed`, `skipped` or
 `aborted`
  - counts of test suites `passed`, `ignored`, `failed` or `skipped`?

 Please note the distinction between `skipped` (was not even started so no
 assertions have been evaluated) and `ignored` (ran to completion with one
 or more failed `BOOST_WARN` assertions).

 AFAIK, test suites cannot be aborted.  Can they?

 In reviewing my ticket, I noticed that `skipped` and `aborted` should
 perhaps be added as test case result values as well.  That way, the report
 includes everything you ever wanted to know about all test cases' and test
 suites' states.

 Adding the output of failed assertions would be nice too, but that's
 outside the scope of this ticket.

-- 
Ticket URL: <https://svn.boost.org/trac/boost/ticket/6129>
Boost C++ Libraries <http://www.boost.org/>
Boost provides free peer-reviewed portable C++ source libraries.