Re: [boost] [Report] 79 regressions on RC_1_34_0 (2007-03-22)

From: Thomas Witt (witt_at_[hidden])
Date: 2007-03-23 00:34:19

Stefan,

Stefan Seefeld wrote:
> Douglas Gregor wrote:
> 
> 
> I didn't find any answer when I asked the last time, so I'm asking again:

FWIW I tried to answer the question before.

> 
> What is the meaning of the absolute number of 'regressions' ? Did this number really go up
> from the last report to the current one ? At least some 'new' ones stem from the inclusion
> of the 'gcc-4.1.1_sunos_i86pc' test run in this report, which wasn't present in the last.

In isolation that number isn't worth much. That being said I am a little 
puzzled by your fixation on this number. I think that the number 
together with the regression tables provides valuable information. It 
just requires some interpretation.

> 
> What determines the test runs that make it into a report ?

Availability. We rely on volunteers to provide test results.

> Is this sunos platform
> really a primary platform for this release ? Why wasn't it tested before ?

Yes. At the point the platform was chosen we had frequent results. 
Testers drop in/out for all kinds of reasons. 
Hardware/vacation/job/configuration it's hard to blame a volunteer for 
any of this, but in the end we have a major reliability and turnaround 
issue.

I would drop the platform right away but in this case we are dealing 
with a general python issue. Dropping the platform won't make it go away.

> 
> How are we ever going to get the number of unexpected failures down to zero ?

Fixing bugs?

> I honestly don't believe it will ever happen, if we continue like that. :-(

Well in some way we already gave up. The only thing we are waiting on is 
the python stuff to get into shape.

> 
> May I suggest to fix a number of 'primary platforms' (and that may well translate
> to specific testers, at this point in the release process), and just disregard anything
> else ?

Personally I think we'll have to do something like this in the future. 
Right now we don't have the infrastructure in place to do this and the 
last thing I want to do at this point is fiddle with regression test 
infrastructure.

Thomas

-- 
Thomas Witt
witt_at_[hidden]