Re: [boost] [EXTERNAL] [testing] Proposal - regression tests results display upgrade

Subject: Re: [boost] [EXTERNAL] [testing] Proposal - regression tests results display upgrade
From: Belcourt, Kenneth (kbelco_at_[hidden])
Date: 2014-08-13 18:53:06

On Aug 13, 2014, at 1:51 PM, Beman Dawes <bdawes_at_[hidden]> wrote:

> On Wed, Aug 13, 2014 at 10:42 AM, Adam Wulkiewicz <adam.wulkiewicz_at_[hidden]
>> wrote:
> 
>> Paul A. Bristow wrote:
>> 
>>> -----Original Message-----
>>>> From: Boost [mailto:boost-bounces_at_[hidden]] On Behalf Of Adam
>>>> 
>>> Wulkiewicz
>>> 
>>>> Would it make sense to use some more descriptive color/naming scheme?
>>>> 
>>>> In particular it would be nice to distinguish between the actual
>>>> failures and
>>>> 
>>> the
>>> 
>>>> situations when the compilation of a test took too much time or an
>>>> output file
>>>> 
>>> was
>>> 
>>>> too big.
>>>> Would it make sense to also distinguish between compilation, linking and
>>>> run
>>>> 
>>> failure?
>>> 
>>> +1  definitely.
>>> 
>>> This is particularly a problem for Boost.Math - the largest library in
>>> Boost, by
>>> far, in both code and tests, with several tests that often time out.
>>> 
>>> Various summary counts of passes and fails would be good as well.
>>> 
>>> It takes a while to scan the current (nice) display, eg
>>> 
>>> http://www.boost.org/development/tests/develop/developer/math.html
>>> 
>> 
>> Ok, I managed to find some time to play with it.
>> AFAIU the reports are generated using the code from:
>> https://github.com/boostorg/boost/tree/develop/tools/regression/src/report,
>> is that correct?
>> 
> 
> Yes, I believe so.
> 
>> 
>> The first change is simple since I'd like to see if I'm doing everything
>> right. I changed the way how those specific fails are displayed. If the
>> compilation fails and at the end of the compiler output (25 last
>> characters) one of the following strings can be found:
>> - "File too big"
>> - "time limit exceeded"
>> the test is considered as "unfinished compilation" and displayed on the
>> library results page as a yellow cell with a link named "fail?". So it's
>> distinguishable from the normal "fail".
>> 
>> Here is the PR: https://github.com/boostorg/boost/pull/25
>> 
> 
> I skimmed it and didn't see any red flags.
> 
> 
>> 
>> I only tested it locally on a test done by 1 runner for Geometry and Math
>> libraries.
>> Is there a way I could test it on results sent by all of the runners?
>> How is this program executed?
> 
> IIRC, Noel Belcourt at sandia runs the tests. Thus he would be a good
> person to merge your pull request since he will likely be the first to
> notice any problems.
> 
> Noel, are you reading this:-?

A little delayed but yes.

> Is there some script which e.g. checks some directory and passes all of
>> the tests as command arguments?
>> 
> He would be the best person to answer that.

I’ll have to investigate, don’t know off the top of my head.

— Noel