Subject: Re: [boost] [testing] Proposal - regression tests results display upgrade
From: Beman Dawes (bdawes_at_[hidden])
Date: 2014-08-13 15:51:47
On Wed, Aug 13, 2014 at 10:42 AM, Adam Wulkiewicz <adam.wulkiewicz_at_[hidden]
> Paul A. Bristow wrote:
>> -----Original Message-----
>>> From: Boost [mailto:boost-bounces_at_[hidden]] On Behalf Of Adam
>>> Would it make sense to use some more descriptive color/naming scheme?
>>> In particular it would be nice to distinguish between the actual
>>> failures and
>>> situations when the compilation of a test took too much time or an
>>> output file
>>> too big.
>>> Would it make sense to also distinguish between compilation, linking and
>> +1 definitely.
>> This is particularly a problem for Boost.Math - the largest library in
>> Boost, by
>> far, in both code and tests, with several tests that often time out.
>> Various summary counts of passes and fails would be good as well.
>> It takes a while to scan the current (nice) display, eg
> Ok, I managed to find some time to play with it.
> AFAIU the reports are generated using the code from:
> is that correct?
Yes, I believe so.
> The first change is simple since I'd like to see if I'm doing everything
> right. I changed the way how those specific fails are displayed. If the
> compilation fails and at the end of the compiler output (25 last
> characters) one of the following strings can be found:
> - "File too big"
> - "time limit exceeded"
> the test is considered as "unfinished compilation" and displayed on the
> library results page as a yellow cell with a link named "fail?". So it's
> distinguishable from the normal "fail".
> Here is the PR: https://github.com/boostorg/boost/pull/25
I skimmed it and didn't see any red flags.
> I only tested it locally on a test done by 1 runner for Geometry and Math
> Is there a way I could test it on results sent by all of the runners?
> How is this program executed?
IIRC, Noel Belcourt at sandia runs the tests. Thus he would be a good
person to merge your pull request since he will likely be the first to
notice any problems.
Noel, are you reading this:-?
> Is there some script which e.g. checks some directory and passes all of
> the tests as command arguments?
He would be the best person to answer that.
Thanks for working that this!