Re: [boost] Regression testing: too many runners?

Subject: Re: [boost] Regression testing: too many runners?
From: John Maddock (jz.maddock_at_[hidden])
Date: 2015-02-08 12:24:03

> Guys,
>
> As you may know, I'm working on automatic regression testing of Boost on
> Android. I spent several weeks debugging it and now it seems to be close
> to the finish - i.e. we'll have soon fully automatic 24x7 regression
> testing with uploading all results to the Boost public FTP. As you can
> see, some results are already uploading on a regular basis -
> http://www.boost.org/development/tests/master/developer/summary.html
> (there are results of just few libraries at the current moment since
> I've limited testing by only those libraries to save time needed for
> debugging testing scripts; as far as I finish with my debugging, I'll
> enable testing of all libraries).
>
> However, I've realized that for thorough testing matrix of the variants
> become really big. Look, right now there are nine runners for Android,
> and this number is in fact very limited. They differs by target ABI
> (three arm variants, x86 and x86_64) and version of Android (API level
> 19 - Android 4.4 and 21 - Android 5.0). However, it has sense to test
> also on Android 4.0, 4.1, 4.2 and 4.3 since theirs market share is still
> large (see https://developer.android.com/about/dashboards/index.html).
> Add to that MIPS target ABI (not yet included to the testing) and
> multiple by two to have tests running with both default settings and
> with -std=c++11 - and you'll get really big total number of runners.
>
> I'm asking for advice from community. It's not the big problem for us to
> run all such tests in all variants, but I'm unsure if it will be
> acceptable for Boost community to see such wide table of runners. I'm
> afraid it will look like a flood. We also publish Android-only results
> on https://boost.crystax.net/master/developer/summary.html and we'll
> definitely display all results from all runners there. Please let me
> know if the same approach would work for
> http://www.boost.org/development/tests/master/developer/summary.html or
> should I limit somehow runners to be uploaded to the Boost FTP.

The only thing I would say, is that if the number of runners is too 
high, there's a danger of developers being drowned in data.  For me the 
most interesting variants would be the machine architechture ones 
(currently most of the tests are being run on Intel), and the least 
interesting would be C++ dialect ones as we already have good coverage 
of that, and I assume there's not much difference between GCC in C++11 
mode on Andriod to GCC on Linux or whatever?

John.