Re: [Boost-testing] [PING] toolset name changing and regressions

From: David Abrahams (dave_at_[hidden])
Date: 2005-08-03 09:00:30

Aleksey Gurtovoy <agurtovoy_at_[hidden]> writes:

> David Abrahams writes:
>> David Abrahams <dave_at_[hidden]> writes:
>>
>>> With all the updating of toolset names we've done recently, have we
>>> got a system in place for ensuring that failures are detected properly
>>> as regressions and not mistaken for failures on newly-added compilers?
>
> Uhm, kind of. The only required toolsets that have been renamed are
> "msvc", "msvc-stlport", and "vc7", and these have been covered (by
> duplicating the corresponding 1.32.0 results under the new names). A
> much more interesting question is what should be our policy regrading
> compiler upgrades, minor or otherwise. For instance, should [brand
> new] CW 9.4 results be reported against CW 8.3 ones from the previous
> release, at least with regard to regressions? How about CW 9.5 vs. CW
> 9.3? 

:)

> I'm inclined to say "yes" at least for the latter, but opitions are
> most welcome.

My feeling is that we need a different color to indicate "the compiler
changed and broke the test or our workaround," because that's a
regression we can't necessarily be responsible for.  We need to be
able to look at our own results and know that we've fixed everything
we *can* be responsible for.  

And the click-through detail should indicate the last version in which
the test passed. :)

-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com