Re: [Boost-testing] Comparing different runs of regression tests

Subject: Re: [Boost-testing] Comparing different runs of regression tests
From: Ben Pope (benpope81_at_[hidden])
Date: 2013-07-02 10:14:30

On 27/06/13 03:13, Marshall Clow wrote:
> Over the last few weeks, I've been collecting the logs of my regression tester (the XML files that get uploaded).
> They include several slightly different configurations:
>
> 	darwin				gcc 4.2.1 + libstdc++ compiling for C++03
> 	clang-darwin			Apple-released clang + libstdc++ compiling for C++03
> 	clang-darwin-tot		Current "tip-of-tree" clang + libstdc++ compiling for C++03
> 	clang-darwin-11		Apple-released clang + libc++ compiling for C++11
> 	clang-darwin-tot11		Current "tip-of-tree" clang + libc++ compiling for C++11
> 	clang-darwin-asan		Current "tip-of-tree" clang + libstdc++ compiling for C++03 using Address Sanitizer
> 	clang-darwin-asan11	Current "tip-of-tree" clang + libc++ compiling for C++11 using Address Sanitizer
>
> Having this data, I started to wonder.
> 	* What are the differences in the results for two different days?
> 		Example: What changed in the test results between Tuesday and Wednesday?
> 	* What are the differences in the results between two different configurations?
> 		Example: What differences are there between using gcc and clang?
> 		Example: What differences are there between C++03 vs. C++11?
> 		Example: What differences are there between "released clang" and "tot-clang"?
> 		Example: What differences are there when you turn on Address Sanitizer?
>
> I've written some python scripts to help answer these questions.
>
> Is this kind of information interesting to anyone besides me?

Yes this is interesting.

I'm just about to define BOOST_THREAD_VERSION=4 for my test runners and 
it would be interesting to see if there is a difference in test results.

Ben