Re: [Boost-testing] out of date release results?

From: Beman Dawes (bdawes_at_[hidden])
Date: 2008-08-05 14:31:59

Eric Niebler wrote:
> Just spoke with Rene about this ...
> 
> Eric Niebler wrote:
>> How come the "report time" on nearly every page of release test 
>> results is dated July 15th?
>>
>> For example:
>> http://www.boost.org/development/tests/release/developer/summary_release.html

AFAIK, that's the wrong page to be looking at. The page I use to make 
decisions is 
http://beta.boost.org/development/tests/release/developer/summary.html

The date on that is Sun, 3 Aug 2008 10:16:33 +0000. That's the last run 
before I shut the machine down for maintenance. It is back up now and 
should produce an updated report in an hour or two.

> We should *all* be checking the release test results, especially the 
> release manager.

I look at the results several times a day during critical times, such as 
when when preparing the beta.

> How did this happen? Here are some hard questions.

No clue. I have no idea how the results on the www.boost.org web site 
get updated from the beta.boost.org web site.

> On what basis did we release the beta?

The beta was released on the basis of a release report that was only a 
few minutes old.

>(That is, do we have release 
> criteria? Written down?)
> Do those criteria have anything to do with the test results?

Of course. The results speak for them; most of the remaining failures 
are minor nits or even false positives.

> Is anybody in charge of the test infrastructure?

Rene works on the scripts, but a lot of the responsibility is 
distributed or ill-defined.

> Does that person look at the test results?
> What can we do to make sure this doesn't happen again?

Take a look at ticket #2150. http://svn.boost.org/trac/boost/ticket/2150

That's one attempt at automatic tools to look for things that have gone 
wrong and report them quickly. It will work on files, from the 
repository or generated by the doc process.

A similar tool that looked at web sites would be a nice QA addition. It 
would check for the presence of specified files and verify their date 
was recent, for a definition of recent specific to each file. Maybe 
check file size, too, or even some content.

> Clearly, we need to delay 1.36 until we can get some fresh test results.

The test results are currently being updated several (up to eight) times 
each day.

> And probably reopen the release branch for bug fixes.

I posted a message several days ago indicating that the release branch 
was open to bug fixes that are stable on trunk.

--Beman