Boost Testing :
From: Beman Dawes (bdawes_at_[hidden])
Date: 2008-08-05 14:31:59
Eric Niebler wrote:
> Just spoke with Rene about this ...
> Eric Niebler wrote:
>> How come the "report time" on nearly every page of release test
>> results is dated July 15th?
>> For example:
AFAIK, that's the wrong page to be looking at. The page I use to make
The date on that is Sun, 3 Aug 2008 10:16:33 +0000. That's the last run
before I shut the machine down for maintenance. It is back up now and
should produce an updated report in an hour or two.
> We should *all* be checking the release test results, especially the
> release manager.
I look at the results several times a day during critical times, such as
when when preparing the beta.
> How did this happen? Here are some hard questions.
No clue. I have no idea how the results on the www.boost.org web site
get updated from the beta.boost.org web site.
> On what basis did we release the beta?
The beta was released on the basis of a release report that was only a
few minutes old.
>(That is, do we have release
> criteria? Written down?)
> Do those criteria have anything to do with the test results?
Of course. The results speak for them; most of the remaining failures
are minor nits or even false positives.
> Is anybody in charge of the test infrastructure?
Rene works on the scripts, but a lot of the responsibility is
distributed or ill-defined.
> Does that person look at the test results?
> What can we do to make sure this doesn't happen again?
Take a look at ticket #2150. http://svn.boost.org/trac/boost/ticket/2150
That's one attempt at automatic tools to look for things that have gone
wrong and report them quickly. It will work on files, from the
repository or generated by the doc process.
A similar tool that looked at web sites would be a nice QA addition. It
would check for the presence of specified files and verify their date
was recent, for a definition of recent specific to each file. Maybe
check file size, too, or even some content.
> Clearly, we need to delay 1.36 until we can get some fresh test results.
The test results are currently being updated several (up to eight) times
> And probably reopen the release branch for bug fixes.
I posted a message several days ago indicating that the release branch
was open to bug fixes that are stable on trunk.