From: Doug Gregor (dgregor_at_[hidden])
Date: 2005-04-01 16:01:54
On Mar 11, 2005, at 10:08 PM, Misha Bergal wrote:
> Douglas Gregor <doug.gregor_at_[hidden]> writes:
>> To improve the reproducibility of results and make testing more
>> predictable, we might want to have the regression scripts always check
>> out using a given date/time tag, e.g., 12:00am EST each night. That
>> way, all of the tests for the day will be on the same codfe. If it
>> helps fix other problems with regression testing, great!
> Doug, I want to put in writing some things which I believe to be
> important to be stated explicitly. I believe that at the end, you are
> the release manager for 1.33.0. As such, you have a great influence on
> what the testing group goals are - you are our "main customer".
I've been a rather silent customer, but time is again (temporarily) my
> You just need to tell us what you consider to be your main problems
> and their priorities.
> For example,
> * Developers/Release manager need to see what changes caused this
> particular test to fail.
I think getting more immediate feedback about new failures would go
along way in helping is determine which changes cause failures. The GCC
developers have a wonderful system that complains very loudly when
someone checks in broken code. When new regressions are found, it:
(1) Determines what code has changed since the last-known-good version
(2) Determines who made changes to the repository since then
(3) E-mails everyone that made changes, giving them a summary of the
new failures and a link to the log file. Alternatively, we could spam
the developer list or a regression-testing list (in decreasing order of
(4) Keeps e-mailing everyone until the problem is fixed.
The last one isn't really necessary, but having something that does the
first three would be *great*. It might even be nice to know when we fix
a regression (it can sometimes be hard to tell give the compiler status
> * Release manager should be able to set what revision of codebase is
> getting tested and get all results for that particular revision
This is extremely important. Being able to easily change all of the
regression testing to a branch means that we might be able to move to a
more lightweight release management process. If it's easy for the
release manager to switch over testing, stability the branch in a week
or two, then release, we'll get releases out without so much fuss.
> * When developer checks something in - she should see the results of
> that checkin ASAP. Something like telling the testing system, test
> me my library on all toolsets and return be the results, quick!
This takes more hardware than we currently have access to, so it's not
very high priority to me.
Boost list run by bdawes at acm.org, gregod at cs.rpi.edu, cpdaniel at pacbell.net, john at johnmaddock.co.uk