Re: [Boost-testing] Issue classification broken

From: Aleksey Gurtovoy (agurtovoy_at_[hidden])
Date: 2006-07-13 05:31:29

David Abrahams writes:
> I'm trying to invest at least a little energy in killing off
> regressions in 1.34, so I look at the unresolved issue report, and I
> see in the report for the parameter library that all squares are
> yellow.  Yellow means either the test or the compiler was newly added.
> But one of the squares is for the "basics" test with borland-5_6_4 and
> http://engineering.meta-comm.com/boost-regression/1_33_1/developer/parameter_release.html
> shows that the "basics" test was passing with borland-5_6_4 in the
> last release.  So what's going on here?

Looking at it, we have two major problems WRT regression reporting:

1. The expected results in "$BOOST_ROOT/libs/expected_results.xml"
   were not updated with 1.33 / 1.33.1 results. IOW the current pages
   display regressions against 1.32, which is why the parameter
   library failures are considered to be "new".

2. The switch to Boost.Build v2 brought in the new toolset naming
   scheme, which, without adjusting toolset names in
   "$BOOST_ROOT/libs/expected_results.xml", breaks the
   regression reporting for all the v2 toolsets.

#1 is fixed and should be reflected in the reports within a few hours,
fixing #2 requires deciding whether we are droping v1 testing
altogether. If we are, we can just adjust the toolset names in
"$BOOST_ROOT/libs/expected_results.xml" and be done with it; if we are
not, the expected results for the renamed toolsets need to be
duplicated, further slowing down the reports update cycle (unless we
implement some sort of toolset aliasing).

-- 
Aleksey Gurtovoy
MetaCommunications Engineering