Re: [boost] [1.35.0] Release criteria compilers

From: Gennadiy Rozental (rogeeff_at_[hidden])
Date: 2007-10-07 17:09:26

Jeff Garland <jeff <at> crystalclearsoftware.com> writes:

> 
> Gennadiy Rozental wrote:
> 
> > 
> > I must say I feel very similar.
> > 
> > I believe Bemans approach is way too pessimistic. And kinda deminishing to 
the 
> > configuration he has no direct access to ;) How VC 8.0 failure is more 
> > critical than let's say gcc on sparc solaris? Assuming that both have 
similar 
> > tester resources. 
> 
> They haven't typically had similar resources.  We've usually had 2-3 VC 
> 8 testers and maybe one gcc on solaris.
> 
> > What I beliee we need is more objective criteria for what is "release 
> > compiler". Specific requirements to number of testers and testing 
turnaround 
> > time comes to mind.

 If one tester with daily turnarond time is agreed upon as "release compiler"  
requirement, they both match the criteria and are in equal positions. IMO the 
more platforms we can report as beeing tested on - the better. 

 You may argue that we test on platform P anyway, but this is not a big help 
for many guys trying to push for boost acceptance, unless this confguration is 
mentions in bold on release notes page. We just need to send clear message 
that it's not promissed that all the libraries works on all tested 
configurations and the compiler status page gives full picture.

> Instead of getting bogged down in trying to define requirements, I think 
> we should simply agree on a the 'primary platform list' (essentially 
> what Beman is doing).  Don't think of this list as release compilers, 
> just a good cross section of the platforms that provide a) good testing 
> support and 

I agree it's simpler. I don't agree it's correct. This list needs constant 
revision and collective decision on which compilers are "more important". This 
is too subjective IMO. For someone, most important compiler is Inter on AIX 
and all other are irrelevant. And percents won't help here either. It's just 
strait road to holy wars.

> b) high standards compliance -- thus minimizing the typical 
> hackery needed to port to this compiler. Again, no compiler/platform is 
> being excluded from testing and we want library authors and people with 
> an interest in a platform to continue to port at their discretion -- but 

I don't believe this is correct criteria at all. The fact that compiler ABC is 
bad doesn't stop us from saying "we tested the release against the compiler 
ABC and here results". In my opinion it's completely orthogonal. From were I 
stand negative results are as good as positive.

> I hope by now we'd all agree that bugging authors of new libs to port to 
> VC6 and holding the release is a bad policy because it harms more users 
> than it helps.

Yes. And I never suggested that. My position is that, following XP guidelines 
any new library is expected to fail everytwhere :0), until it start works 
somewhere. It's regressions we need to pay some attentions to. And to some 
degree as well. If it become unreasonably difficult so failures are frequent 
we drop the configuration even if it has enough testing resources. Let's say 
the compiler A on platform P failes more than 50% of tests (including expected 
failures), than we might decide that regressions detected at release date 
doesn't worth fixing and we drop the configuration instead.
 
> >   Additionally it's important to split new failures from regression. New 
> > failures we just mark as expected on day of release and ignore. At the 
> 
> Well, this doesn't quite work for me.  If a new library can't pass tests 
> on the 'primary platform list' then it needs to be removed from the 
> release because it's not ready.

As I mentioned before, in my opinion it's completely irrelevant. Library is 
accepted during review, not by release manager. It's not his call to decide 
which compilers the library should support. Let's say tommorow we accept the 
library that employs rvalue references. It passes tests only on some 
experimental compiler we don't test against for release. I don't see it as 
showstopper. We mark the library tests as expected to fail on all compilers 
for now and move on.
 Release manager should stop fighting for "world peace". In other words it's 
not his job to make sure library passes the tests. This responceability lies 
on the library author. His job is to ensure backward compartibility.

> > beggining of each release we can cleanup "expected failure" status from 
the 
> > test and it becomes new failure again(unless it fixed of cource). 
Regressions 
> > does need to be fixed before release. And here the only place were we may 
go 
> > into grey area. If library A is failing on platform P, while it was 
working 
> > before, what should we do? I personally believe we don't have many cases 
like 
> > this.
> >   What I can propose is some kind of tolerance based approach:
> >   1) If platform P has more than V1% of failures - we name platform P as 
> > unsupported
> >   2) if library failes on more than V2% of platforms - we revert the 
library.
> >   3) Otherwise we name the test as expected and add into special section 
of 
> > release notes (saying this test for the library A is now failing on 
platform 
> > P). 
> > 
> > Values V1 and V2 we can agree upon. 
> 
> I don't really object to the idea except that we all know it's more 
> complicated because of library dependencies.  For example, date-time is 
> failing on the trunk right now -- it's source is unchanged since 1.34 -- 
> so it's some other change that has broken some of the regressions.  You 
> can't revert date-time because it hasn't changed.  We need to track down 
> the change that's breaking it and revert...of course, that might break 
> some new feature some other lib depends on.  The bottom line here is 
> that check-ins that break other libraries won't be tolerated for long...

It's all true, but IMO irrelevant to the subject. Dependency handling and 
deciding what needs to be rolled is not simple (at best). But this is the 
issue we are facing irrespective which compilers are used for release.
 
> > Note that all this can and should be done in one shot on day of release. 
No 
> > second chances guys ;)
> 
> I don't think it can work like that, sorry.

I do not see why yet. We do need to esteblish mechanism for rolling back 
library (along with all the libraries it depends on and that depends on it), 
but I think it should be rare event, which, however complicated, can and 
should be done in one shot.
 
> Jeff

Gennadiy