[Boost-testing] Comparing different runs of regression tests

Subject: [Boost-testing] Comparing different runs of regression tests
From: Marshall Clow (mclow.lists_at_[hidden])
Date: 2013-06-26 15:13:45

Over the last few weeks, I've been collecting the logs of my regression tester (the XML files that get uploaded).
They include several slightly different configurations:

        darwin				gcc 4.2.1 + libstdc++ compiling for C++03
        clang-darwin			Apple-released clang + libstdc++ compiling for C++03
        clang-darwin-tot		Current "tip-of-tree" clang + libstdc++ compiling for C++03
        clang-darwin-11		Apple-released clang + libc++ compiling for C++11
        clang-darwin-tot11		Current "tip-of-tree" clang + libc++ compiling for C++11
        clang-darwin-asan		Current "tip-of-tree" clang + libstdc++ compiling for C++03 using Address Sanitizer
        clang-darwin-asan11	Current "tip-of-tree" clang + libc++ compiling for C++11 using Address Sanitizer

Having this data, I started to wonder.
        * What are the differences in the results for two different days?
                Example: What changed in the test results between Tuesday and Wednesday?
        * What are the differences in the results between two different configurations?
                Example: What differences are there between using gcc and clang?
                Example: What differences are there between C++03 vs. C++11?
                Example: What differences are there between "released clang" and "tot-clang"?
                Example: What differences are there when you turn on Address Sanitizer?

I've written some python scripts to help answer these questions.

Is this kind of information interesting to anyone besides me?

-- Marshall