Re: [boost] [testing] Proposal - regression tests results display upgrade

Subject: Re: [boost] [testing] Proposal - regression tests results display upgrade
From: Adam Wulkiewicz (adam.wulkiewicz_at_[hidden])
Date: 2014-06-05 14:21:03

Paul A. Bristow wrote:
>
>> -----Original Message-----
>> From: Boost [mailto:boost-bounces_at_[hidden]] On Behalf Of Adam
> Wulkiewicz
>> Sent: 28 May 2014 14:09
>> To: boost_at_[hidden]
>> Subject: [boost] [testing] Proposal - regression tests results display upgrade
>>
>> Hi,
>>
>> Currently test failures of a new feature are always marked with the yellow
> color and
>> a word 'fail'.
>> Would it make sense to use some more descriptive color/naming scheme?
>> In particular it would be nice to distinguish between the actual failures and
> the
>> situations when the compilation of a test took too much time or an output file
> was
>> too big.
>> Would it make sense to also distinguish between compilation, linking and run
> failure?
>> E.g. instead of a simple 'fail' use 'fail (c)', 'fail (l)'
>> and 'fail (r)'.
>> E.g. the color for actual failures could be orange and for "others"
>> could stay yellow.
>>
>> Furthermore, currently it's impossible to see the results of compilation if a
> test
>> passed.
>> Would it make sense to change it?
>> This would allow us to view and fix warnings even if the test was passed.
>> The test passed with warnings could be marked e.g. with 'pass (w)' or the
> color could
>> be set to some between yellow and this light green.
> +1  definitely.
>
> This is particularly a problem for Boost.Math - the largest library in Boost, by
> far, in both code and tests, with several tests that often time out.
>
> Various summary counts of passes and fails would be good as well.
>
> It takes a while to scan the current (nice) display, eg
>
> http://www.boost.org/development/tests/develop/developer/math.html
>

I thought about some list of phrases/regexps which if found/matched in 
the compiler output or even the output of the regression tool would 
result in various coloring of cells and names.
E.g. something like:

fail    yellow    "compile with /bigobj"
fail    yellow    "300 second time limit exceeded"
fail(c) orange    ^"Compile".+"fail"$
# ...

Or something similar in XML.
Could also be set per-library if needed.

As much as I'd like to play with it I'm affraid I won't find enough time.

Regards,
Adam