From: David Abrahams (dave_at_[hidden])
Date: 2004-02-13 07:46:52
Aleksey Gurtovoy <agurtovoy_at_[hidden]> writes:
> David Abrahams writes:
>> Martin Wille <mw8329_at_[hidden]> writes:
>>> Hartmut Kaiser wrote:
>>>> What about generating an email to the maintainers of the
>>>> corresponding library, which makes sure they will know about a
>>>> failed test ... :-)
>>> I often had failures due to problems with my local
> Same here.
>>> Emails about failed tests should not be sent
>>> without being checked by a human before. (Which main-
>>> tainer would want to be notified about my disk-full
>> You ;-)
>> Seriously, I'm sure there's a way to deal with this sort of thing,
> Probably, but it needs to be figured out, since in our experience
> misconfiguration/setup issues arise often enough to significantly
> impact "worthiness" of automatic notifications.
> Hmm, may be the first step of figuring it out would be to collect and
> classify such cases. "Disk full" is definitely the first one to be
> registered (we had it here as well).
>> becausqe the BuildBot is being used successfully by several projects.
> That doesn't necessarily mean that they solved the issue, though.
> FWIW, it's not mentioned in the paper.
As a matter of fact, Brian emailed me to say that automatic
notifications aren't implemented yet:
---- > The way BuildBot works, it first notifies the perpetrator of the > offending checkin. After that, if the problem isn't fixed, it can > make more-urgent notifications to a different/wider audience. It can > use any number of notification methods including email and IRC. FYI, that feature hasn't been implemented yet. There are plans for it (by more than just me), but we haven't yet done the work. At the moment, failing builds are indicated in two ways, both of which require polling: the status web page shows red, and has links to let you see the test failures the optional IRC bot will respond to queries about build status, letting you know that the latest build is "failed" instead of "passed" The "notify the perpetrator" part is easy, as we already have a list of CVS names whose changes are involved in each build. Email and IRC are easy, it's just a question of registering a mapping between CVS names and email addresses / irc nicknames. The "if the problem isn't fixed" part is trickier, as it requires the ability to recognize that two successive test runs have the same failure mode (we need to parse the test case results and identify which test cases are failing). I've found that notification is more effective when the test suite is usually clean. It is easy for a project to get into a mode where there are one or two difficult tests which "always fail", and after a week or two in that state it is hard to attain the motivation to fix them. In that mode, constant "the test suite is still broken" reminders may do more harm than good. The notification scheme needs to take this (human) property into account. In addition, the possibility of intermittent test failures (like the minor timing problems we get on the Twisted tests all the time, usually due to changes in system load) means that we need to be able to recognize them and verify the failure (by re-running the failed test) before harassing anybody. If not, I believe we run the risk of irritating developers to the point of ignoring the buildbot results, which then reduces the point of installing it in the first place. That's why I have (personally) prioritized "push-mode" notification *after* better test-case parsing. cheers, -Brian -- Dave Abrahams Boost Consulting www.boost-consulting.com
Boost list run by bdawes at acm.org, gregod at cs.rpi.edu, cpdaniel at pacbell.net, john at johnmaddock.co.uk