Re: [boost] Boost regression notification (2005-12-05 [RC_1_33_0])

From: Daryle Walker (darylew_at_[hidden])
Date: 2005-12-11 02:41:01

On 12/7/05 2:18 PM, "Stefan Seefeld" <seefeld_at_[hidden]> wrote:

> Daryle Walker wrote:
>> On 12/6/05 6:00 PM, "Stefan Seefeld" <seefeld_at_[hidden]> wrote:
>>> Daryle Walker wrote:
>> [SNIP a library that failed on every compiler]
>>>> This could be another case to watch out for: when a library fails on every
>>>> compiler (or a least a lot of them).  Here it would be the library that's
>>>> possibly broken.
>>> 
>>> Isn't that exactly what these tests are supposed to measure ? Or do you mean
>>> that in case *all* tests fail the report could just mention a single
>>> failure, on a different granularity scale ?
>> 
>> Yes, I meant the latter.  If we suddenly see a compiler fail on every
>> library, or a library fail on every compiler, then we probably have a
>> configuration issue.  (But how do we implement "suddenly," since the tests
>> don't keep a history of past results?  We only want to block blooper runs,
>> not compilers or libraries that legitimately fail everything.)
> 
> Indeed. One thing I already proposed in the past: some dummy tests (akin
> to autoconf macros) that make sure a sane configuration / environment is
> available at all. These tests could cover a specific toolchain, or even
> some lightweight library-specific code. If these are known as dependencies
> for the rest of the test suite(s), quite a bit of computing power (and
> mail bandwidth) could be spared whenever these fail.

Maybe we can go by percentages too.  If a test fail a super-majority of the
compilers, then we could report that test has a configuration problem.  We
could do a similar test for a library if a super-majority of its tests fail.
A compiler would get reported if it appears in a super-majority of the tests
as a failure.  I don't know which percentage to use (66%, 75%, 90%, etc.).

-- 
Daryle Walker
Mac, Internet, and Video Game Junkie
darylew AT hotmail DOT com