[boost] Regression testing: too many runners?

Subject: [boost] Regression testing: too many runners?
From: Dmitry Moskalchuk (dm_at_[hidden])
Date: 2015-02-07 15:15:47

Guys,

As you may know, I'm working on automatic regression testing of Boost on
Android. I spent several weeks debugging it and now it seems to be close
to the finish - i.e. we'll have soon fully automatic 24x7 regression
testing with uploading all results to the Boost public FTP. As you can
see, some results are already uploading on a regular basis -
http://www.boost.org/development/tests/master/developer/summary.html
(there are results of just few libraries at the current moment since
I've limited testing by only those libraries to save time needed for
debugging testing scripts; as far as I finish with my debugging, I'll
enable testing of all libraries).

However, I've realized that for thorough testing matrix of the variants
become really big. Look, right now there are nine runners for Android,
and this number is in fact very limited. They differs by target ABI
(three arm variants, x86 and x86_64) and version of Android (API level
19 - Android 4.4 and 21 - Android 5.0). However, it has sense to test
also on Android 4.0, 4.1, 4.2 and 4.3 since theirs market share is still
large (see https://developer.android.com/about/dashboards/index.html).
Add to that MIPS target ABI (not yet included to the testing) and
multiple by two to have tests running with both default settings and
with -std=c++11 - and you'll get really big total number of runners.

I'm asking for advice from community. It's not the big problem for us to
run all such tests in all variants, but I'm unsure if it will be
acceptable for Boost community to see such wide table of runners. I'm
afraid it will look like a flood. We also publish Android-only results
on https://boost.crystax.net/master/developer/summary.html and we'll
definitely display all results from all runners there. Please let me
know if the same approach would work for
http://www.boost.org/development/tests/master/developer/summary.html or
should I limit somehow runners to be uploaded to the Boost FTP.

-- 
Dmitry Moskalchuk