Subject: Re: [boost] [all][testing] Regression summary upgrades
From: Adam Wulkiewicz (adam.wulkiewicz_at_[hidden])
Date: 2015-05-13 10:47:54
Paul A. Bristow wrote:
> -----Original Message-----
>> From: Boost [mailto:boost-bounces_at_[hidden]] On Behalf Of Adam Wulkiewicz
>> E.g. I'm thinking about upgrading the Summary page
>> to also display specific failure type, i.e. the most significant one.
>> Right now it's all yellow.
>> Would it be useful for someone?
>> The hierarchy could be:
>> comp > link > run > fail > cerr > time > file
> Yellow is a bit discouraging, for example for Boost.Math where the details are much more
> encouragingly green.
Exactly. After seing it I personally get an impression that Boost is not
working in general, which of course is not true.
> Is it easily possible to show the fraction of tests and/or platforms that pass?
> (However I accept that the requirements are different for libraries will zillions of tests compared
> to those with few).
For now I prepared a PR adding the above proposal:
If it's merged we'll see if it helps and how much.
When the summary is generated all of the tests from all libraries are
checked anyway. This is done to generate the statistics shown on the top
of the Summary page (Unusable, Regressions and New failures). And later
when the table itself is generated. So, it wouldn't be a problem to
gather some additional statistics for a library per toolset. But I don't
have a clear view what should be presented and how. Well I have a few ideas:
1. In a cell put a percent of failing tests with a color of the type of
the most significant one. I don't know if this would be clear enough
because the numbers are in general hard to analyse on a first sight.
2. Instead of the numbers there could be some character representation
of the percent of the tests that are failures like:
or something similar.
3. Use some wierd unicode characters for the indicator. Or explicitly
write fraction as: ¼, ½, ¾.
4. Or more graphically. The cells for failures probably could be divided
into 2 colors vertically. The height of the yellow/orange bar inside a
cell could indicate the percentage of the failures (with 4 possible
heights: <=25%, <=50%, <=75% and <=100%). With this we could put the
name of the most significant failure in the cell. This could be nice
actually but it's hard to predict how clear would it be. It requires
Do you have some specific idea?
Boost list run by bdawes at acm.org, gregod at cs.rpi.edu, cpdaniel at pacbell.net, john at johnmaddock.co.uk