Re: [boost] [Report] 79 regressions on RC_1_34_0 (2007-03-22)

From: David Abrahams (dave_at_[hidden])
Date: 2007-03-22 16:09:07

on Thu Mar 22 2007, Stefan Seefeld <seefeld-AT-sympatico.ca> wrote:

> David Abrahams wrote:
>> on Thu Mar 22 2007, Stefan Seefeld <seefeld-AT-sympatico.ca> wrote:
>> 
>>> I didn't find any answer when I asked the last time, so I'm asking again:
>>>
>>> What is the meaning of the absolute number of 'regressions' ? 
>> 
>> Tells us how many regressions there are?
>
> What is the reference point ? The last report ? The last release ?

It's supposed to be the last release, IIUC.

>>> Did this number really go up
>>> from the last report to the current one ? 
>> 
>> I don't know.
>
> But this is at the heart of the question: If the simple addition of
> a new test run changes the number of failures, how can this number
> possibly represent regressions ? Does anybody even know what test
> runs were accounted for at the point the last release was done ?

IIUC, yes.

> What I'm trying to say is that, instead of just picking up whatever
> results are found by the report generator, an explicit list of
> test runners should be chosen as 'primary platforms' to be able
> to measure progress, and extrapolate into the future.
>
> Right now it is hard to tell whether there was any progress at
> all over the last couple of months.

Yep.

-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com