|
|
Boost Testing : |
Subject: Re: [Boost-testing] Comparing different runs of regression tests
From: Marshall Clow (mclow.lists_at_[hidden])
Date: 2013-07-02 14:51:40
On Jun 29, 2013, at 3:11 PM, Jim Bell <Jim_at_[hidden]> wrote:
>
>
>
> On 2013-06-26 2:13 PM, Marshall Clow wrote:
>> Over the last few weeks, I've been collecting the logs of my regression tester (the XML files that get uploaded).
>> ...
>> Having this data, I started to wonder.
>> * What are the differences in the results for two different days?
>> Example: What changed in the test results between Tuesday and Wednesday?
>> * What are the differences in the results between two different configurations?
>> Example: What differences are there between using gcc and clang?
>> Example: What differences are there between C++03 vs. C++11?
>> Example: What differences are there between "released clang" and "tot-clang"?
>> Example: What differences are there when you turn on Address Sanitizer?
>>
>> I've written some python scripts to help answer these questions.
>>
>> Is this kind of information interesting to anyone besides me?
>>
>
> I've thought it would be very interesting to see a a per-platform delta (new passes/fails from last regression). That ties to a pretty narrow SVN r-range, which I think would be very helpful.
>
> Can it go on the website somewhere?
Putting the python scripts up is no problem; I'll push them somewhere (like github), and people can take them and run with them.
There are practical problems with "putting them on the website", since they need regression logs to work from, and those are fairly large.
[ My trunk runner (6 configs) is about 300 MB/day ]
If anyone has ideas on how to deploy this, I'll be happy to work with them.
-- Marshall
Marshall Clow Idio Software <mailto:mclow.lists_at_[hidden]>
A.D. 1517: Martin Luther nails his 95 Theses to the church door and is promptly moderated down to (-1, Flamebait).
-- Yu Suzuki