I've currently been running three testing machines for linux, with several testers on each, (windows is completely separate, and I'm happy with how that is running)...here was my past strategy:

teeks99-03 - develop - lots of compiler versions (all I could get to run on ubuntu 12.04: gcc-4.4-4.8, clang-3.0-3.4)

teeks99-04 - develop - only two compilers, with fast turnaround times (gcc-4.8, clang-3.4)

teeks99-05 - master - lots of compiler versions (all I could get to run on ubuntu 12.04: gcc-4.4-4.8, clang-3.0-3.4)

I can try to set things up like this again, but there are two reasons I was thinking about changing:

1) It seems that a lot of the gcc/clang versions have the same pass/fail results as the other versions of that compiler, so we're not adding a ton (though we may be adding the crucial points where a compiler change breaks something...is this something that devs have seen in practice?)

2) The new version of my runners (Ubuntu 14.04) doesn't easily support the old gcc versions, it would take a lot of effort to get some of the old ones running. Clang is even worse...it is much more difficult to get even two versions running side by side now.

Instead of running lots of different versions, I was thinking about running various compatibility options of the two main compilers...I've seen some other test runners with things like libc++,  c++11, c++14, etc.  My real question, would developers be better served by these options than different versions? Would they prefer both with longer revisit times between each test type? Other thoughts?

Tom