Re: [boost] [1.35.0] Release criteria compilers

From: Jeff Garland (jeff_at_[hidden])
Date: 2007-10-07 03:54:37

Gennadiy Rozental wrote:

> 
> I must say I feel very similar.
> 
> I believe Bemans approach is way too pessimistic. And kinda deminishing to the 
> configuration he has no direct access to ;) How VC 8.0 failure is more 
> critical than let's say gcc on sparc solaris? Assuming that both have similar 
> tester resources. 

They haven't typically had similar resources.  We've usually had 2-3 VC 
8 testers and maybe one gcc on solaris.

> What I beliee we need is more objective criteria for what is "release 
> compiler". Specific requirements to number of testers and testing turnaround 
> time comes to mind.

Instead of getting bogged down in trying to define requirements, I think 
we should simply agree on a the 'primary platform list' (essentially 
what Beman is doing).  Don't think of this list as release compilers, 
just a good cross section of the platforms that provide a) good testing 
support and b) high standards compliance -- thus minimizing the typical 
hackery needed to port to this compiler. Again, no compiler/platform is 
being excluded from testing and we want library authors and people with 
an interest in a platform to continue to port at their discretion -- but 
I hope by now we'd all agree that bugging authors of new libs to port to 
VC6 and holding the release is a bad policy because it harms more users 
than it helps.

>   Additionally it's important to split new failures from regression. New 
> failures we just mark as expected on day of release and ignore. At the 

Well, this doesn't quite work for me.  If a new library can't pass tests 
on the 'primary platform list' then it needs to be removed from the 
release because it's not ready.

> beggining of each release we can cleanup "expected failure" status from the 
> test and it becomes new failure again(unless it fixed of cource). Regressions 
> does need to be fixed before release. And here the only place were we may go 
> into grey area. If library A is failing on platform P, while it was working 
> before, what should we do? I personally believe we don't have many cases like 
> this.
>   What I can propose is some kind of tolerance based approach:
>   1) If platform P has more than V1% of failures - we name platform P as 
> unsupported
>   2) if library failes on more than V2% of platforms - we revert the library.
>   3) Otherwise we name the test as expected and add into special section of 
> release notes (saying this test for the library A is now failing on platform 
> P). 
> 
> Values V1 and V2 we can agree upon. 

I don't really object to the idea except that we all know it's more 
complicated because of library dependencies.  For example, date-time is 
failing on the trunk right now -- it's source is unchanged since 1.34 -- 
so it's some other change that has broken some of the regressions.  You 
can't revert date-time because it hasn't changed.  We need to track down 
the change that's breaking it and revert...of course, that might break 
some new feature some other lib depends on.  The bottom line here is 
that check-ins that break other libraries won't be tolerated for long...

> Note that all this can and should be done in one shot on day of release. No 
> second chances guys ;)

I don't think it can work like that, sorry.

Jeff