Re: [Boost-testing] Comparing different runs of regression tests

Subject: Re: [Boost-testing] Comparing different runs of regression tests
From: Marshall Clow (mclow.lists_at_[hidden])
Date: 2013-07-09 00:22:21

On Jul 2, 2013, at 7:14 AM, Ben Pope <benpope81_at_[hidden]> wrote:

> On 27/06/13 03:13, Marshall Clow wrote:
>> Over the last few weeks, I've been collecting the logs of my regression tester (the XML files that get uploaded).
>> They include several slightly different configurations:
>> 
>> 	darwin				gcc 4.2.1 + libstdc++ compiling for C++03
>> 	clang-darwin			Apple-released clang + libstdc++ compiling for C++03
>> 	clang-darwin-tot		Current "tip-of-tree" clang + libstdc++ compiling for C++03
>> 	clang-darwin-11		Apple-released clang + libc++ compiling for C++11
>> 	clang-darwin-tot11		Current "tip-of-tree" clang + libc++ compiling for C++11
>> 	clang-darwin-asan		Current "tip-of-tree" clang + libstdc++ compiling for C++03 using Address Sanitizer
>> 	clang-darwin-asan11	Current "tip-of-tree" clang + libc++ compiling for C++11 using Address Sanitizer
>> 
>> Having this data, I started to wonder.
>> 	* What are the differences in the results for two different days?
>> 		Example: What changed in the test results between Tuesday and Wednesday?
>> 	* What are the differences in the results between two different configurations?
>> 		Example: What differences are there between using gcc and clang?
>> 		Example: What differences are there between C++03 vs. C++11?
>> 		Example: What differences are there between "released clang" and "tot-clang"?
>> 		Example: What differences are there when you turn on Address Sanitizer?
>> 
>> I've written some python scripts to help answer these questions.
>> 
>> Is this kind of information interesting to anyone besides me?
> 
> Yes this is interesting.
> 
> I'm just about to define BOOST_THREAD_VERSION=4 for my test runners and it would be interesting to see if there is a difference in test results.

Ben -- 

I'll just attach the current versions here (they're quite small) and set them up in source control later.
Please let me know what you think they ought to do differently.

[ And no, I don't hold them up as shining examples of software engineering, either - if you know better ways to accomplish what they do, please let me know  ]

-- Marshall

Marshall Clow     Idio Software   <mailto:mclow.lists_at_[hidden]>

A.D. 1517: Martin Luther nails his 95 Theses to the church door and is promptly moderated down to (-1, Flamebait).
        -- Yu Suzuki