Re: [boost] Re: [1.33.0] Let's start preparations...

From: Doug Gregor (dgregor_at_[hidden])
Date: 2005-04-01 16:01:54

On Mar 11, 2005, at 10:08 PM, Misha Bergal wrote:

> Douglas Gregor <doug.gregor_at_[hidden]> writes:
>
>> To improve the reproducibility of results and make testing more
>> predictable, we might want to have the regression scripts always check
>> out using a given date/time tag, e.g., 12:00am EST each night. That
>> way, all of the tests for the day will be on the same codfe. If it
>> helps fix other problems with regression testing, great!
>
> Doug, I want to put in writing some things which I believe to be
> important to be stated explicitly. I believe that at the end, you are
> the release manager for 1.33.0. As such, you have a great influence on
> what the testing group goals are - you are our "main customer".

I've been a rather silent customer, but time is again (temporarily) my 
friend :)

> You just need to tell us what you consider to be your main problems
> and their priorities.
>
> For example,
>
> * Developers/Release manager need to see what changes caused this
>   particular test to fail.

I think getting more immediate feedback about new failures would go 
along way in helping is determine which changes cause failures. The GCC 
developers have a wonderful system that complains very loudly when 
someone checks in broken code. When new regressions are found, it:
   (1) Determines what code has changed since the last-known-good version
   (2) Determines who made changes to the repository since then
   (3) E-mails everyone that made changes, giving them a summary of the 
new failures and a link to the log file. Alternatively, we could spam 
the developer list or a regression-testing list (in decreasing order of 
effectiveness).
   (4) Keeps e-mailing everyone until the problem is fixed.

The last one isn't really necessary, but having something that does the 
first three would be *great*. It might even be nice to know when we fix 
a regression (it can sometimes be hard to tell give the compiler status 
tables).

> * Release manager should be able to set what revision of codebase is
>   getting tested and get all results for that particular revision

This is extremely important. Being able to easily change all of the 
regression testing to a branch means that we might be able to move to a 
more lightweight release management process. If it's easy for the 
release manager to switch over testing, stability the branch in a week 
or two, then release, we'll get releases out without so much fuss.

> * When developer checks something in - she should see the results of
>   that checkin ASAP. Something like telling the testing system, test
>   me my library on all toolsets and return be the results, quick!

This takes more hardware than we currently have access to, so it's not 
very high priority to me.

        Doug