|
|
Boost Testing : |
From: Aleksey Gurtovoy (agurtovoy_at_[hidden])
Date: 2005-06-01 14:39:14
David Abrahams writes:
> "Aleksey Gurtovoy" <agurtovoy_at_[hidden]> writes:
>
>>> This sort of capability is one thing that we're hoping to get out of
>>> BuildBot.
>>
>> Please stop saying that. It's a red herring. BuildBot by itself is
>> uncapable to report _regressions_. To do that, the results need to be
>> processed against some previous state, test case by test case. After you
>> _do_ identify the failure as a regression (as opposite to an ever
>> failing compiler or a new library/test case check in), change-driven
>> rebuilds/testing will often help to localize the source of the
>> disturbance more precisely, that's for sure. But regression notifications
>> per se have no relationship to change-driven rebuilds and the other way
>> around.
>
> 'Scuse me. When I first found out about BuildBot the guy giving the
> talk spent a while on the topic of regression notifications. That
> feature later turned out not to have been implemented yet, but I have
> assumed there was progress on it in the past 2 years. No?
They "regression notifications" are _failure_ notifications. There is
a huge difference for us. If you look at the summary page, you'll see
that currently we have 4248 new failures + 1807 test cases marked as
unusable (most of which fail as well), and "just" 206 regressions.
>
>> Implementing each of these two requires its own investment, and
>> saying the opposite doesn't really help our cause.
>
> I didn't say anything to the contrary. I only said it was one of the
> things we were hoping to get out of BuildBot. That's true for me, at
> least.
But we are not going to, and I pointed this out before. When you say
that we are, people who otherwise would be motivated to look into this
are likely to lose their interest ("OK, it's pending something bigger
that somebody else's already working on"), which is quite hurting
because we don't have many of them in the first place.
-- Aleksey Gurtovoy MetaCommunications Engineering