[boost] [Regression Reports] Test case categorization

From: Aleksey Gurtovoy (agurtovoy_at_[hidden])
Date: 2005-05-11 23:31:00

I'd like to draw library authors' attention to a recent generalization
in the explicit markup scheme which allows one to control the grouping
of a particular library's test cases in the Boost-wide regression
reports, as demostrated here --
http://www.meta-comm.com/engineering/boost-regression/example/developer/library_00-00.html
(http://tinyurl.com/bdd5u).

The generalization is aimed at improving representation of library
reports in at least two common situations:

1) "Duplicate" test cases: many libraries have two or more variations
   of the same test case for different configurations (DLL vs. static
   linking, char vs. wchar_t, etc.).

2) Test cases covering corner-case/optional/advanced functionality.

In both of these cases, it's common to have an explicit markup (or
actual failures) painting a group of test cases in a different color
for one or more deficient toolsets -- see, for example, Boost.Thread
page (http://tinyurl.com/8duf6). Test case categorization allows
library authors to reduce the amount of visual clutter in the reports,
giving them a more organized/structured look -- ultimately improving
visitors' experience.

Assigning a category to a test case or a group of test cases is just a
matter of specifying the 'category' attribute to the corresponding
'<test>' elements in "status/explicit-failures-markup.xml", e.g.

    <library name="my_library">
        <test name="*_dll" category="DLL"/>
    </library>

or

    <library name="iterator">
        <test name="is_convertible_fail" category="Corner-case tests">
            <mark-failure>
                <toolset name="borland"/>
                <toolset name="borland-5_6_4"/>
                <toolset name="msvc"/>
                <toolset name="vc7"/>
                <note refid="2"/>
            </mark-failure>
            ...

As always, feedback is more than welcome.

Enjoy,
-- 
Aleksey Gurtovoy
MetaCommunications Engineering