Re: [boost] [all][testing] Regression summary upgrades

Subject: Re: [boost] [all][testing] Regression summary upgrades
From: Adam Wulkiewicz (adam.wulkiewicz_at_[hidden])
Date: 2015-05-13 10:47:54

Paul A. Bristow wrote:
>   -----Original Message-----
>> From: Boost [mailto:boost-bounces_at_[hidden]] On Behalf Of Adam Wulkiewicz
>> E.g. I'm thinking about upgrading the Summary page
>> http://www.boost.org/development/tests/develop/developer/summary.html
>> to also display specific failure type, i.e. the most significant one.
>> Right now it's all yellow.
>> Would it be useful for someone?
>> The hierarchy could be:
>> comp > link > run > fail > cerr > time > file
> Yellow is a bit discouraging, for example for Boost.Math where the details are much more
> encouragingly green.

Exactly. After seing it I personally get an impression that Boost is not 
working in general, which of course is not true.

> Is it easily possible to show the fraction of tests and/or platforms that pass?
>
> (However I accept that the requirements are different for libraries will zillions of tests compared
> to those with few).

For now I prepared a PR adding the above proposal:
https://github.com/boostorg/regression/pull/16
If it's merged we'll see if it helps and how much.

When the summary is generated all of the tests from all libraries are 
checked anyway. This is done to generate the statistics shown on the top 
of the Summary page (Unusable, Regressions and New failures). And later 
when the table itself is generated. So, it wouldn't be a problem to 
gather some additional statistics for a library per toolset. But I don't 
have a clear view what should be presented and how. Well I have a few ideas:

1. In a cell put a percent of failing tests with a color of the type of 
the most significant one. I don't know if this would be clear enough 
because the numbers are in general hard to analyse on a first sight.

2. Instead of the numbers there could be some character representation 
of the percent of the tests that are failures like:
[|||  ]
[---  ]
or something similar.

3. Use some wierd unicode characters for the indicator. Or explicitly 
write fraction as: ¼, ½, ¾.

4. Or more graphically. The cells for failures probably could be divided 
into 2 colors vertically. The height of the yellow/orange bar inside a 
cell could indicate the percentage of the failures (with 4 possible 
heights: <=25%, <=50%, <=75% and <=100%). With this we could put the 
name of the most significant failure in the cell. This could be nice 
actually but it's hard to predict how clear would it be. It requires 
some testing.

Do you have some specific idea?

Regards,
Adam