[boost] Additional Test Configurations

Subject: [boost] Additional Test Configurations
From: Tom Kent (lists_at_[hidden])
Date: 2016-06-07 07:54:21

About a month ago I got three new machines dedicated to testing boost
online (thanks to Boost Community/Steering Committee/Conservancy!). These
three machines are currently running tests along with a couple other
machines that I have. Now that things are up and running, I'm looking for
some community input as to how best utilize them.

Here's the current setup:

teeks99-01 - Ubuntu 14.04 + Docker
 - gcc-4.4 (master + develop) Docker precise + ubuntu-toolchain-r/test
 - gcc-4.5 (master + develop) Docker precise + ubuntu-toolchain-r/test
 - gcc-4.6 (master + develop) Docker precise + ubuntu-toolchain-r/test
 - gcc-4.7 (master + develop) Docker precise + ubuntu-toolchain-r/test
 - gcc-4.8 (master + develop) Docker trusty + ubuntu-toolchain-r/test
 - gcc-4.9 (master + develop) Docker trusty + ubuntu-toolchain-r/test
 - gcc-5 (master + develop) Docker trusty + ubuntu-toolchain-r/test
 - clang-3.0 (master + develop) Docker trusty + h-rayflood/llvm
 - clang-3.1 (master + develop) Docker trusty + h-rayflood/llvm
 - clang-3.2 (master + develop) Docker trusty + h-rayflood/llvm
 - clang-3.3 (master + develop) Docker trusty + ubuntu-toolchain-r/test
 - clang-3.4 (master + develop) Docker trusty + ubuntu-toolchain-r/test
 - clang-3.5 (master + develop) Docker trusty + ubuntu-toolchain-r/test
 - clang-3.6 (master + develop) Docker trusty
 - clang-3.7 (master + develop) Docker trusty + llvm.org/apt
teeks99-02 - Ubuntu 16.04 + Docker
 - gcc-4.4 (master + develop) Docker precise + ubuntu-toolchain-r/test
 - gcc-4.5 (master + develop) Docker precise + ubuntu-toolchain-r/test
 - gcc-4.6 (master + develop) Docker precise + ubuntu-toolchain-r/test
 - gcc-4.7 (master + develop) Docker precise + ubuntu-toolchain-r/test
 - gcc-4.8 (master + develop) Docker trusty + ubuntu-toolchain-r/test
 - gcc-4.9 (master + develop) Docker trusty + ubuntu-toolchain-r/test
 - gcc-5 (master + develop) Docker trusty + ubuntu-toolchain-r/test
 - clang-3.0 (master + develop) Docker trusty + h-rayflood/llvm
 - clang-3.1 (master + develop) Docker trusty + h-rayflood/llvm
 - clang-3.2 (master + develop) Docker trusty + h-rayflood/llvm
 - clang-3.3 (master + develop) Docker trusty + ubuntu-toolchain-r/test
 - clang-3.4 (master + develop) Docker trusty + ubuntu-toolchain-r/test
 - clang-3.5 (master + develop) Docker trusty + ubuntu-toolchain-r/test
 - clang-3.6 (master + develop) Docker trusty
 - clang-3.7 (master + develop) Docker trusty + llvm.org/apt
teeks99-03 - Ubuntu 16.04 + Docker
 - gcc-5 (develop) Docker trusty + ubuntu-toolchain-r/test
 - clang-3.7 (develop) Docker trusty + llvm.org/apt
teeks99-05 - Ubuntu 14.04 (Azure VM)
 - gcc-4.8 (master)
 - clang-3.4 (master)
teeks99-08 - Windows 2012 (KVM VM)
 - msvc-8.0 (master + develop) SP1
 - msvc-9.0 (master + develop) SP1
 - msvc-10.0 (master + develop) SP1
 - msvc-11.0 (master + develop) Update 4
 - msvc-12.0 (master + develop) Update 5
 - msvc-14.0 (master + develop) Update 2
teeks99-09 - Windows 2012 (KVM VM)
 - msvc-8.0 (master + develop) SP1
 - msvc-9.0 (master + develop) SP1
 - msvc-10.0 (master + develop) SP1
 - msvc-11.0 (master + develop) Update 4
 - msvc-12.0 (master + develop) Update 5
 - msvc-14.0 (master + develop) Update 2

I'm currently working on getting gcc-6 and clang 3.8, sorry they aren't in
the list yet. I'm planning to retire the teeks99-01 shortly and converting
the ancient teeks99-05 to something newer, still doing boost testing.

All these configurations and the scripts that keep them cycling are on
github at:
https://github.com/teeks99/boost-build
For each platform there are a couple configurations that I try to run
frequently to give 'rapid' results as developers are committing code. The
configurations for windows are set so that every few runs it will run the
msvc-12.0 (develop) test, so that is never more than ~10hrs old. Similarly
on linux, the gcc-5 and clang-3.7 (develp) tests get run every few hours.
Just to give a feeling of the timing, Linux runs take .75-3 hrs and Windows
runs take 1.5-4hrs, depending on compiler, os, hardware, etc. All these
configurations have minimal compiler options set. MSVC + GCC builds have no
options set, clang has `<cxxflags>-Wno-c99-extensions`.

>From a lot of the discussing I've seen on the lists, it seems that there's
a lot of work going into C++ 11/14/1z development. I was thinking it would
be good to add some of those switches. Which ones? How many?

Baseline Standard
-std=c++98 (I believe this is the default for all the g++/clang++ instances
above)
-std=c++11/0x
-std=c++14/1y
-std=c++1z

GNU Standard
-std=gnu++98
-std=gnu++11/0x
-std=gnu++14/1y
-std=gnu++1z

Clang Library
-stdlib=libc++
-stdlib=libstdc++

Optimization
-O2?

Warnings
-Wall
-Wextra

Others?

I'm guessing that most of these will be of interest to at least a small
subset of developers. There probably isn't a need to apply all possible
combinations to all possible compilers however. (Not to mention that it
would make the results matrix even more overwhelming.)
Which combinations are important for you to see in the matrix? What do you
think should be the 'rapid' configurations?

I'm also thinking about additional compilers (in addition to
gcc-6/clang-3.9 mentioned above), though these will take a bit of work on
my part, so will only get implemented as I have time available.
- gcc HEAD from GIT
- clang HEAD from GIT
- cygwin/gcc
- mingw
- clang on windows
- Other?

Thanks in advance for your input!
Tom