Re: [boost] [1.35.0] Release criteria compilers

From: Gennadiy Rozental (rogeeff_at_[hidden])
Date: 2007-10-07 01:58:49

Peter Dimov <pdimov <at> pdimov.com> writes:

> 
> Jeff Garland:
> 
> > Let me put it another way.  We will continue testing and fixing on the
> > all compilers/platforms for which we have volunteers.  The difference
> > is, that we aren't going to hold the release up to ensure that less
> > standard compliant compilers are all green.   By reducing the number of
> > platforms we can reduce the cycle time for developers to be sure changes
> > are working.
> 
> I agree that dropping compilers that do not provide rapid testing turnaround 
> will improve matters, but I'm not seeing the connection with the level of 
> standard compliance or the number of the release platforms. The tests run in 
> parallel, so 1 day is 1 day. Some platforms may prove problematic for other 
> reasons, of course, but we can and should deal with that if and when it 
> happens, not preemptively. 

I must say I feel very similar.

I believe Bemans approach is way too pessimistic. And kinda deminishing to the 
configuration he has no direct access to ;) How VC 8.0 failure is more 
critical than let's say gcc on sparc solaris? Assuming that both have similar 
tester resources. 

What I beliee we need is more objective criteria for what is "release 
compiler". Specific requirements to number of testers and testing turnaround 
time comes to mind.

  Additionally it's important to split new failures from regression. New 
failures we just mark as expected on day of release and ignore. At the 
beggining of each release we can cleanup "expected failure" status from the 
test and it becomes new failure again(unless it fixed of cource). Regressions 
does need to be fixed before release. And here the only place were we may go 
into grey area. If library A is failing on platform P, while it was working 
before, what should we do? I personally believe we don't have many cases like 
this.
  What I can propose is some kind of tolerance based approach:
  1) If platform P has more than V1% of failures - we name platform P as 
unsupported
  2) if library failes on more than V2% of platforms - we revert the library.
  3) Otherwise we name the test as expected and add into special section of 
release notes (saying this test for the library A is now failing on platform 
P). 

Values V1 and V2 we can agree upon. 

Note that all this can and should be done in one shot on day of release. No 
second chances guys ;)

Gennadiy