[boost] [all][testing] Regression summary upgrades

Subject: [boost] [all][testing] Regression summary upgrades
From: Adam Wulkiewicz (adam.wulkiewicz_at_[hidden])
Date: 2015-05-11 11:46:46

Hi,

Recently the regression summary was upgraded. The cells corresponding to 
the failing tests can have names comp, link, run or fail.
This way we can see what's the nature of the failure without going into 
details.
See, e.g. the tests for Math:
http://www.boost.org/development/tests/develop/developer/math.html

The last category of failures (generic "fail") is for the others or unknown.
Currently the "Lib" errors are falling into this category. There are 
some in Math, e.g.:
http://www.boost.org/development/tests/develop/developer/output/CrystaX-NET-apilevel-19-armeabi-v7a-hard-boost-bin-v2-libs-math-test-powm1_sqrtp1m1_test-test-clang-linux-3-5-release-link-static-target-os-android-threading-multi.html
So we could mark them explicitly as "lib" failures and then the 
other/unknown failures could be marked as "unkn" or left as it is now - 
"fail".
This has sense if there can be other "types" of failures, e.g. caused by 
some error in the testing scripts, Build or Regression, for which the 
reason is somehow unknown. Though I haven't seen them.

I propose to go further with this and to mark the compilation failures 
for which we know the reason of the failure in a special way:
- "file" - file too big (reported on Windows, e.g. 
http://www.boost.org/development/tests/develop/developer/output/MinGW-w64-4-8%20jc-bell-boost-bin-v2-libs-intrusive-test-unordered_multiset_test-test-gcc-mingw-4-8-2-debug.html)
- "time" - time limit exceeded (e.g. 
http://www.boost.org/development/tests/develop/developer/output/teeks99-03b-Ubuntu12-04-64-boost-bin-v2-libs-math-test-test_carlson-test-gcc-4-5-debug-link-static.html)
I saw them in many libraries and I think it would be very useful for 
libraries having many tests like Math or Geometry.

The other types of failures I could think of aren't that important for 
me personally but maybe someone could appreciate them, e.g. the 
compilers developers:
- "ierr" or "cerr" - internal compiler error or segmentation fault 
during compilation (e.g. 
http://www.boost.org/development/tests/develop/developer/output/PNNL-RHEL6_64-boost-bin-v2-libs-math-test-multiprc_concept_check_1-test-gcc-4-4-release-link-static.html 
or 
http://www.boost.org/development/tests/develop/developer/output/teeks99-03h-Ubuntu12-04-64-boost-bin-v2-libs-math-test-tools_roots_inc_test-test-clang-linux-3-2-debug-link-static.html)
- "segf" - segmentation fault during run 
(http://www.boost.org/development/tests/develop/developer/output/BenPope%20x86_64%20Ubuntu-boost-bin-v2-libs-math-test-common_factor_test-test-clang-linux-3-7~msan~c14_libc++-debug-link-static.html)

Furthermore I propose to use various colors for different failure reasons.

You can see the examples here:
https://github.com/awulkiew/summary-enhancer
https://github.com/awulkiew/summary-enhancer/tree/master/example/pages

Of course the colors could be changed. We should be able to modify them 
in the master.css file. In the examples above they are varying much to 
make a clear distinction between them. But if you like the yelow collor 
then all of the new failures could be yelowish with a slight accent of 
another color, e.g. compilation errors could be yellow with orange 
accent and "time"/"file" errors with green accent.

Regards,
Adam