Re: [boost] Experimental audience-targeted regression results

From: Peter Dimov (pdimov_at_[hidden])
Date: 2003-06-19 09:40:32

Aleksey Gurtovoy wrote:
> Peter Dimov wrote:
>
>> The summaries are nice, but the red "broken" thing on the user page
>> may be too intimidating,
>
> When will show the actual status, it shouldn't be (it doesn't yet,
> since
> some cooperation from library authors is needed - please see below).
> Or, rather, if it still is, then it's the status that is too
> intimidating, not the report :).

I'm not sure I agree here. The problem is that the user summary says:

    Red status basically means that you won't be able to use the library.

This is often simply not true.

You do provide me with the tools to eliminate a false red status, but this
is a "guilty unless proven otherwise" approach; pretty much everyone can
easily run the regression tests on various broken configurations and it is
up to me to hunt down every "non-critical" failure _and_ extract the
relevant toolset name. In the meantime users have already rejected the
library on the basis of that little red rectangle.

Note also that Beman's intel-win32 toolset passed shared_ptr_test but your
intel-win32 toolset did not, and I can't distinguish the two in
expected_results.xml.

In short, I think that this approach will result in more relaxed, "common
denominator" tests, where any failure indeed "basically means that you won't
be able to use the library". A failure in shared_ptr_test (but not in
smart_ptr_test) _usually_ means that there are some obscure corner cases
where this compiler/configuration is buggy. A failure in bind_test usually
means that you may or may not be able to use bind depending on the
complexity of the bind expression and the phase of the moon _but_ if it
compiles it typically runs fine. ;-)

I'm not sure how this can be avoided, but - a hypothetical example - if a
library passes 29 out of its 30 tests, "BROKEN" simply doesn't seem
appropriate. I'd also like to see some kind of per-test tagging ("brokenness
weight") and not per-toolset tagging as the toolset name is unreliable.

A way for the test executable to signal "only non-critical tests failed" may
help, too.

I realize I'm basically reiterating what you wrote in your earlier
message... I guess the reason is that the system as it stands doesn't really
accomplish its goals, or I may be misunderstanding how it is supposed to
work.

The per-library developer summary is great, though. ;-)