RE: [boost] Experimental audience-targeted regression results

From: Aleksey Gurtovoy (agurtovoy_at_[hidden])
Date: 2003-06-19 08:56:02

Rene Rivera wrote:
> [2003-06-18] Aleksey Gurtovoy wrote:
> 
> >.... as per 
> http://article.gmane.org/gmane.comp.lib.boost.devel/20648 are
> >available from here:
> >
> > * user summary page -
> >http://boost.sourceforge.net/regression-logs/user_summary_page.html
> > * developer summary page -
> >http://boost.sourceforge.net/regression-logs/developer_summar
> y_page.html
> >
> >Please comment!
> 
> On content...
> 
> The summary at the library level is good. But it has one drawback. 
> If I looked at that table as a user, I would give up in short order
> to use Boost at all. It just looks like it's mostly broken. 

Partially it's due to a misconfigured 'como-win32' toolset setup, and
partially because at the moment MPL is the only library with specified
expected failures. As more and more libraries will have those, the report
will reflect the actual status quo. 

> So 
> having what is essentially a binary indicator is misleading. 

As long as it reports things correctly, it's not.

> Parhaps 
> an indication of how much works and doesn't is more informative to a 
> user (and developer). Knowing the difference between 1 failure vs. 
> 50 failures is most times more important. After all many times as a 
> user I may not need the funcitonality that is failing. And having 
> just a works/fails discourages further investigation as to what parts 
> do and don't work.

See http://article.gmane.org/gmane.comp.lib.boost.devel/20648 for the
discussion on this exact topic.

> The results page makes more sense to me than the summary page. Having 
> the seemingly binary indicator here is OK.
> 
> On HTML/style...
> 
> You really need to work on the HTML. The result pages do strange 
> overlapping text in some browsers. If you expect the background and 
> text colors to be a certain way, PLEASE put that in the HTML. My 
> non-white default background makes the pages look less than 
> flattering.

Yeah, can imagine :). We'll fix these in the next revision.

> 
> ...Indicators of various kinds:
> 
> Don't use background colors as indicators. It just obscure any 
> possible information that the text is trying to indicate. 

The first thing I would say is that this and most of other points below
are highly subjective. I would appreciate if we discuss them in a less 
imperative tone ;).

Basically, there is a single (and simple!) idea that both motivates 
using of background colors and suggests the particular scheme, that is, 
that you should be able to determine the status of things by just 
glancing over them. You don't have to read (and, ideally, to scroll 
anything). It's especially important and wanted for the CVS health 
(developer) report, which normally should be a full green field.
We are not pioneers here - see Built Bot 
(http://twistedmatrix.com/~warner.twistd/) or Mozilla reports, - nor we
think we are picking up a bad practice.

> And to say nothing about color-blind contention. Stick to using 
> background colors to delineate sections, or in place of rulers (as in 
> table borders).
> 
> If you see the need to use an indicator other than text stick to using
> shapes. For example you could replace the red/green indicators with X 
> and + icons, respectively. Using color on the icons then has the same 
> effect you have now but without the clutter, and without overly relying 
> on color.
> 
> Be more informative in the text indicators. Using "OK" and "OK*" as
> different indicators just looses any meaning that each may have on 
> thei own. They both look just the same to me. Suggestion use "Supported" 
> and "Partial". 

They are close to be the same from a user standpoint, but I like your 
suggestion. Only those are too long :(.

> Even though you did use different terms for the non-working indicator 
> in the user page, the ones you chose are again equivalent; "doesn't work" 
> has the same meaning as "broken". 

IMO it doesn't, and the legend tries to explain the difference.

> Pick something that conveys the intended meaning better. In this case: 
> "Unsupported"/"Fails" seems more appropriate IMO.

Not bad, too, but again, too long. It's important to keep the table 
compact. Any other suggestions?

> 
> The developer summary has one serious problem. You have two OK 
> indicators for different things. The dark green OK is just wrong. 

Nope, it's not :).

> If a test is supposed to not-succeed having it succeed is a failure. 
> Or is there some other meaning you intended? 

A test will be marked as dark green if it was known to fail on the 
particular toolset, given up upon, specified as a known failure 
(probably even critical one), and then suddenly it starts working.

[...]

> 
> ...Table headers/side bars:
> 
> Unless the number of libraries get's really large, there's no point in
> having the column labels at the bottom of the table.

It's already large enough to not fit into my screen. Even if you have 
a huge monitor, it doesn't hurt having those, does it?

> Repeating the library/toolset column on the right is more debatable as 
> to it's need. Splitting the table in half, when needed, and either 
> arranging them side by side or one after the other might be an option. 
> The center alignment on the library/toolset labels is distracting, it 
> just makes reading it harder, especially if you are trying to scan for 
> a specific name.

We can try the left alignment and see how it works.

> That alignment also applies to the cell content. Sticking to the language
> standards (English) for this makes it easier on the reader.

Strongly disagree, here. I definitely want to see the status centered.

Thanks for the comments,
Aleksey