[boost] Re: 1.31.0 regressions

From: David Abrahams (dave_at_[hidden])
Date: 2004-02-13 07:46:52

Aleksey Gurtovoy <agurtovoy_at_[hidden]> writes:

> David Abrahams writes:
>> Martin Wille <mw8329_at_[hidden]> writes:
>>
>>> Hartmut Kaiser wrote:
>>>
>>>> What about generating an email to the maintainers of the
>>>> corresponding library, which makes sure they will know about a
>>>> failed test ... :-)
>>>
>>> I often had failures due to problems with my local
>>> setup. 
>
> Same here.
>
>>> Emails about failed tests should not be sent
>>> without being checked by a human before. (Which main-
>>> tainer would want to be notified about my disk-full
>>> errors?)
>>
>> You ;-)
>>
>> Seriously, I'm sure there's a way to deal with this sort of thing,
>
> Probably, but it needs to be figured out, since in our experience
> misconfiguration/setup issues arise often enough to significantly
> impact "worthiness" of automatic notifications.
>
> Hmm, may be the first step of figuring it out would be to collect and
> classify such cases. "Disk full" is definitely the first one to be
> registered (we had it here as well).
>
>> becausqe the BuildBot is being used successfully by several projects.
>
> That doesn't necessarily mean that they solved the issue, though.
> FWIW, it's not mentioned in the paper.

As a matter of fact, Brian emailed me to say that automatic
notifications aren't implemented yet:

----
> The way BuildBot works, it first notifies the perpetrator of the
> offending checkin.  After that, if the problem isn't fixed, it can
> make more-urgent notifications to a different/wider audience.  It can
> use any number of notification methods including email and IRC.
FYI, that feature hasn't been implemented yet. There are plans for it (by
more than just me), but we haven't yet done the work.
At the moment, failing builds are indicated in two ways, both of which
require polling:
 the status web page shows red, and has links to let you see the test failures
 the optional IRC bot will respond to queries about build status, letting you
 know that the latest build is "failed" instead of "passed"
The "notify the perpetrator" part is easy, as we already have a list of CVS
names whose changes are involved in each build. Email and IRC are easy, it's
just a question of registering a mapping between CVS names and email
addresses / irc nicknames. The "if the problem isn't fixed" part is trickier,
as it requires the ability to recognize that two successive test runs have
the same failure mode (we need to parse the test case results and identify
which test cases are failing).
I've found that notification is more effective when the test suite is usually
clean. It is easy for a project to get into a mode where there are one or two
difficult tests which "always fail", and after a week or two in that state it
is hard to attain the motivation to fix them. In that mode, constant "the
test suite is still broken" reminders may do more harm than good. The
notification scheme needs to take this (human) property into account.
In addition, the possibility of intermittent test failures (like the minor
timing problems we get on the Twisted tests all the time, usually due to
changes in system load) means that we need to be able to recognize them and
verify the failure (by re-running the failed test) before harassing anybody.
If not, I believe we run the risk of irritating developers to the point of
ignoring the buildbot results, which then reduces the point of installing it
in the first place. That's why I have (personally) prioritized "push-mode"
notification *after* better test-case parsing.
cheers,
 -Brian
-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com