Re: [Boost-testing] interpreting test results

Subject: Re: [Boost-testing] interpreting test results
From: Stefan Seefeld (stefan_at_[hidden])
Date: 2016-03-07 15:25:50

On 07.03.2016 11:28, Steven Watanabe wrote:
> AMDG
>
> On 03/07/2016 06:26 AM, Stefan Seefeld wrote:
>> * Clicking on a specific cell (i.e., a 'fail' of a particular test), I'm
>> taken to
>> http://www.boost.org/development/tests/develop/developer/CrystaX-apilevel-21-armeabi-v7a-gnu-libstdc++-python-gcc-4-9-andreas_beyer-variants_.html,
>> which contains "Output by test variants:", followed by two
>> (build-variant-specific) links. Both of these take me to pages
>> containing "fail" and another link, which then contains "succeed".
>> Nowhere do I see an actual failure message. Is this intentional ? How am
>> I to interpret these data ??
>>
>   It's not intentional.  It's a bug that appears
> when a library fails to build.  process_jam_log
> assumes that every target has a single source,
> so when there is more than one, it ends up picking
> randomly.

Well, I have to admit that at present
http://www.boost.org/development/tests/develop/developer/python.html is
unusable for me. I basically have to (try to) ignore all columns but
those that are "mostly green", as only those may contain "real" failures.

I think having a way to clearly differentiate failed tests from failed
(library) builds is a very important usability requirement. Even better
if by some simple criteria failed builds could be excluded from this
test result matrix. (It might still be useful to see the build logs for
those failed library builds, but definitely not as part of the test
results matrix.)

        Stefan

-- 
      ...ich hab' noch einen Koffer in Berlin...