Re: [boost] Enhancing regression tests summary

Subject: Re: [boost] Enhancing regression tests summary
From: Adam Wulkiewicz (adam.wulkiewicz_at_[hidden])
Date: 2014-10-31 15:29:20

Beman Dawes wrote:
> On Fri, Oct 31, 2014 at 11:31 AM, Rene Rivera <grafikrobot_at_[hidden]> wrote:
>
>> On Fri, Oct 31, 2014 at 10:18 AM, Robert Ramey <ramey_at_[hidden]> wrote:
>>
>>> Adam Wulkiewicz wrote
>>>> Hi,
>>>>
>>>> Some time ago I was writing about regression summaries enhancements,
>>>> that I'd be nice to see more meaningful list of errors, etc. I asked
>>>> questions, even prepared a PR for regression, to no avail.
>>>>
>>>> So I developed a simple program downloading summaries, test logs and
>>>> saving the enhanced result.
>>>> You may check it out here:
>> https://github.com/awulkiew/summary-enhancer
>>> Very nice - but rather than downloading and enhancing the results,
>> wouldn't
>>> it
>>> be simpler to just enhance the current program.  That is clone the the
>>> current results display tool, incorporate the enhancements, and then
>>> merge the changes into the current tool?
>>>
>> +1
>>
> +1
>

Of course it would be simpler. In fact I proposed this some time ago 
(http://boost.2283326.n4.nabble.com/testing-Proposal-regression-tests-results-display-upgrade-td4662953.html). 
But in practice there wasn't enough feedback/help.

At that Point I had some questions about the way how summary pages are 
generated. Which tools are used exactly, what parameters are passed etc. 
I found C++ code of a tool generating summary pages, modified it and 
tested on locally generated XMLs (by the run.py). I prepared a PR 
(https://github.com/boostorg/boost/pull/25) adding simple enhancements 
to check if my locally-tested program will work with the official setup. 
I have no access to the machine on which the pages are generated or even 
to the actual files sent by the runners so can't do any real testing.

Even though I wasn't sure if this C++ generator is the one that had been 
used to generate the official summary pages, I wanted to have something 
that actually works. In fact I suspect that they're generated by the 
XSLT. It's because there are some attributes on summary pages that 
couldn't be generated by the C++ tool. On the other hand they can be 
found in the XSLT files.

Furthermore, I suspect that summary pages for develop and master 
branches may be generated somehow differently because for some libraries 
the pages for master are missing (e.g. Geometry, Spirit) but this is a 
wild guess.

So finally I decided to implement something myself, something that would 
be useful for me and anyone interested but didn't require anyone's 
involvement. In the same time I wanted to develop a proof of concept 
showing what I have in mind. So if you have any ideas how to move this 
forward and add this functionality to the officially used tool, I'm open 
to suggestion.

Regards,
Adam