Boost logo

Boost :

From: Douglas Gregor (gregod_at_[hidden])
Date: 2002-02-11 11:55:12

On Monday 11 February 2002 12:35 am, you wrote:
> The biggest problem I have with the approach is that it appears to assume
> success as the default. Most developers have at most a few
> compilers/platforms available, and can only reasonably support those to
> which they have access. It actually seems to me that an expectation of
> success or failure is orthogonal to supported-ness. For example, we can say
> that type_traits supports MSVC6, but has many expected failures on that
> platform.

I guess I see supportness as a documentation issue only. If a library states
that is is supported on platform X, then we can expect bugs on that platform
to be fixed in a reasonably amount of time and we can expect reasonably good
behavior on that platform.

Expected failures are for cases where there is a justifiable problem with the
compiler/platform that won't or can't be fixed. Remember that testing on a
new compiler/platform should work well for a library, if that
compiler/platform conforms closely to the standard. If we expect failure on a
new compiler, valid problems with the Boost libraries may not be found because
  1) the author might use a very permissive compiler
  2) we expected all the testcases to fail on ConformC++ 10.5, so nobody ever
looked at the errors

> I think the requirement for release should be that tests give the expected
> results (for both successes and failures) on all platform/compiler
> combinations supported by the corresponding library. Of course we'll test
> on other platform/compiler combinations. A library supported over a
> reasonable spectrum is likely to work in many more contexts than the
> supported ones.
> -Dave

This last sentence of yours - it seems to imply that we should assume success
as the default?

In any case, I agree that the release requirement should be that the tests
give the expected results. The set of testcases that is expected to fail, for
any given compiler/platform, still needs to be a part of the regression
testing system. Look at some regression testing output - either the dashboard
or the compiler status tables - and you'll see a bunch of red for failures.
If there is a set of failures there already, what's one more failure? Just a
drop in the bucket, and it'll be ignored. This is especially the case with
the dashboard, because once there are any errors (one failing
library can cause 100 compiler messages), how would anyone notice that their
last checkin caused it to go to 101? With expected failures as part of the
regression testing system, even in the presence of failures the dashboard can
be green - zero errors. Then when you perform a checkin and the dashboard
turns red, it's harder to let that slip.


Boost list run by bdawes at, gregod at, cpdaniel at, john at