[Boost-testing] interpreting test results

Subject: [Boost-testing] interpreting test results
From: Stefan Seefeld (stefan_at_[hidden])
Date: 2016-03-07 08:26:00

Hello,

I'm trying to use the test results posted online to get a grasp of the
state of a library (and specifically, whether a recent check-in has
fixed a previous bug, or caused a regression. However, I find this to be
quite a daunting task, for multiple reasons, so I want to see whether
I'm missing anything, or should use a different approach. Here are some
issues I observe:

* the amount of data posted on
http://www.boost.org/development/tests/develop/developer/summary.html is
huge, so even just to find the relevant bits is non-trivial

* more specifically, when I look at
http://www.boost.org/development/tests/develop/developer/python.html, I
find mostly failures (the vast majority of cells a yellow). Is that
really representative of the state of Boost.Python on the develop branch ??

* It seems more likely that rather than the tested Boost code being that
buggy, there are fundamental issues with the setup of the tester. So
instead of flagging all the tests as "failed", it would be more useful
to have some simple "meta-tests" that test the testing environment
itself, and then just don't display the test results unless those
meta-tests have validated the tester.

* Clicking on a specific cell (i.e., a 'fail' of a particular test), I'm
taken to
http://www.boost.org/development/tests/develop/developer/CrystaX-apilevel-21-armeabi-v7a-gnu-libstdc++-python-gcc-4-9-andreas_beyer-variants_.html,
which contains "Output by test variants:", followed by two
(build-variant-specific) links. Both of these take me to pages
containing "fail" and another link, which then contains "succeed".
Nowhere do I see an actual failure message. Is this intentional ? How am
I to interpret these data ??

* Columns on the test matrix correspond to individual builds, which are
often a bit dated. It might be useful to have some logic (javascript ?)
to mask test results prior to a given date / revision.

* It would also be useful to know the schedule of the builds, to know
how long to wait after a checkin before test results corresponding to
that will show up.

Thanks,
        Stefan

-- 
      ...ich hab' noch einen Koffer in Berlin...