Re: [Boost-testing] What web views do we need?

From: David Abrahams (dave_at_[hidden])
Date: 2007-12-11 10:29:59

on Tue Dec 11 2007, Aleksey Gurtovoy <agurtovoy-AT-meta-comm.com> wrote:

> David Abrahams wrote:
>> Neither the current Bitten output nor Boost's current standard
>> regression testing tables provide an optimal UI for most Boosters'
>> purposes.
>>
>> Rather that jumping right into thinking about UI details, I thought
>> I'd open a discussion of what we need to be able to quickly learn by
>> visiting one or more testing webpages.  I've started a list of
>> high-level questions (at the level of libraries, not individual test
>> cases) below; hopefully with your participation we can come up with a
>> fairly complete list, which I can post on a wiki page for reference
>> while we do actual page design.
>>
>> I may be jumping the gun slightly, but as I write this down, it seems
>> to boil down to only a few distinct questions for a given revision or
>> revision range:
>>
>> * Which unexpected failures were introduced and/or removed?
>
> Introduced and/or removed since the previous revision, or something
> else?

Introduced by the repository revision (range) in question -- see below

>> * Which library/platform combinations have unexpected failures?
>
> ... and regressions. Sure.

I thought this through, and it seems to me that there are only two
cases: 

1. The new failure has not been marked expected, in which case it will
   show up as unexpected

2. The new failure has been marked expected, in which case,
   presumably, someone thinks there's a good reason for the failure
   and it's not a cause for concern.

I see a few marginal uses for the information "it wasn't failing in
the last release but is now."  For example, if a test was labelled as
a regression despite also beging marked expected, one might scrutinize
the markup a little more carefully to make sure we really mean it.
These cases, however, do not seem compelling enough to make "is it a
regression?" an important question for us.

I know it sounds wierd to ignore regressions in regression testing;
maybe I'm missing something.

>> * Which library/platform combinations have not yet been tested?
>> * Which of those combinations had unexpected failures when last tested?
>
> Sure.
>
>>
>> We would want to be able to filter these results to:
>>
>> * a set of libraries
>> * a set of platforms, including an easy way to say "release platforms"
>> * a repository revision range, including an easy way to say "HEAD"
>>
>> For example, a natural "overall health of Boost" display might be
>> filtered to:
>>
>> * all libraries
>> * all release platforms
>> * the latest repository revision
>>
>> You could find out about the health of your own libraries by filtering
>> to that set.
>>
>> Have I missed anything?
>
> All the questions above are centered around failure tracking, which is
> certainly one of the most important workflows the report pages should
> facilitate, but it's not the only workflow.
>
> You also need a "regular" view of the codebase health, showing all the
> libraries with their status, whatever it is. 

Just to make sure they're actually being tested at all?

> The "failures" view(s) then can be a filter on top of a regular
> view, similarly/in addition to the other criteria you list.

Or a sort order criterion.

--
Dave Abrahams
Boost Consulting
http://www.boost-consulting.com