Re: [boost] Release Candidate Time

From: Douglas Gregor (gregod_at_[hidden])
Date: 2002-02-11 11:55:12

On Monday 11 February 2002 12:35 am, you wrote:
> The biggest problem I have with the approach is that it appears to assume
> success as the default. Most developers have at most a few
> compilers/platforms available, and can only reasonably support those to
> which they have access. It actually seems to me that an expectation of
> success or failure is orthogonal to supported-ness. For example, we can say
> that type_traits supports MSVC6, but has many expected failures on that
> platform.

I guess I see supportness as a documentation issue only. If a library states 
that is is supported on platform X, then we can expect bugs on that platform 
to be fixed in a reasonably amount of time and we can expect reasonably good 
behavior on that platform. 

Expected failures are for cases where there is a justifiable problem with the 
compiler/platform that won't or can't be fixed. Remember that testing on a 
new compiler/platform should work well for a library, if that 
compiler/platform conforms closely to the standard. If we expect failure on a 
new compiler, valid problems with the Boost libraries may not be found because
  1) the author might use a very permissive compiler
  2) we expected all the testcases to fail on ConformC++ 10.5, so nobody ever 
looked at the errors

> I think the requirement for release should be that tests give the expected
> results (for both successes and failures) on all platform/compiler
> combinations supported by the corresponding library. Of course we'll test
> on other platform/compiler combinations. A library supported over a
> reasonable spectrum is likely to work in many more contexts than the
> supported ones.
>
> -Dave

This last sentence of yours - it seems to imply that we should assume success 
as the default?

In any case, I agree that the release requirement should be that the tests 
give the expected results. The set of testcases that is expected to fail, for 
any given compiler/platform, still needs to be a part of the regression 
testing system. Look at some regression testing output - either the dashboard 
or the compiler status tables - and you'll see a bunch of red for failures. 
If there is a set of failures there already, what's one more failure? Just a 
drop in the bucket, and it'll be ignored. This is especially the case with 
the dashboard, because once there are any errors (one failing 
library can cause 100 compiler messages), how would anyone notice that their 
last checkin caused it to go to 101? With expected failures as part of the 
regression testing system, even in the presence of failures the dashboard can 
be green - zero errors. Then when you perform a checkin and the dashboard 
turns red, it's harder to let that slip.

        Doug