[Boost-testing] Linux regression test runners - What do devs want?

Subject: [Boost-testing] Linux regression test runners - What do devs want?
From: Tom Kent (lists_at_[hidden])
Date: 2015-01-03 11:36:10

I've been pondering this recently, but a forced upgrade of my azure VMs
last night is pressing the issue...and I wanted to solicit feedback before
I go and set things back up...

I've currently been running three testing machines for linux, with several
testers on each, (windows is completely separate, and I'm happy with how
that is running)...here was my past strategy:

teeks99-03 - develop - lots of compiler versions (all I could get to run on
ubuntu 12.04: gcc-4.4-4.8, clang-3.0-3.4)
teeks99-04 - develop - only two compilers, with fast turnaround times
(gcc-4.8, clang-3.4)
teeks99-05 - master - lots of compiler versions (all I could get to run on
ubuntu 12.04: gcc-4.4-4.8, clang-3.0-3.4)

I can try to set things up like this again, but there are two reasons I was
thinking about changing:
1) It seems that a lot of the gcc/clang versions have the same pass/fail
results as the other versions of that compiler, so we're not adding a ton
(though we may be adding the crucial points where a compiler change breaks
something...is this something that devs have seen in practice?)
2) The new version of my runners (Ubuntu 14.04) doesn't easily support the
old gcc versions, it would take a lot of effort to get some of the old ones
running. Clang is even worse...it is much more difficult to get even two
versions running side by side now.

Instead of running lots of different versions, I was thinking about running
various compatibility options of the two main compilers...I've seen some
other test runners with things like libc++,  c++11, c++14, etc.  My real
question, would developers be better served by these options than different
versions? Would they prefer both with longer revisit times between each
test type? Other thoughts?

Tom