Re: [boost] Linux regression test runners - What do devs want?

Subject: Re: [boost] Linux regression test runners - What do devs want?
From: Andrey Semashev (andrey.semashev_at_[hidden])
Date: 2015-01-03 18:53:53

First, I'd like to thank you for running tests for so many
configurations. It is most helpful.

On Sat, Jan 3, 2015 at 7:36 PM, Tom Kent <lists_at_[hidden]> wrote:
> I've been pondering this recently, but a forced upgrade of my azure VMs
> last night is pressing the issue...and I wanted to solicit feedback before
> I go and set things back up...

[snip]

> I can try to set things up like this again, but there are two reasons I was
> thinking about changing:
> 1) It seems that a lot of the gcc/clang versions have the same pass/fail
> results as the other versions of that compiler, so we're not adding a ton
> (though we may be adding the crucial points where a compiler change breaks
> something...is this something that devs have seen in practice?)

This is true that developer errors tend to influence multiple, if not
all compilers. However, this kind of errors is also the easiest to
discover and fix. Often the developer is able to reproduce such
failure locally.

The other kind of problems is compiler bugs. These are naturally
specific to the compiler versions, sometimes up to the patch version.
The developer often does not have all compiler versions installed, so
this kind of problems are hard to discover and fix locally. For this
reason having multiple compiler versions in the matrix is invaluable
help to developers, and I would very much like to have it. From
personal experience, I do stumble upon compiler-specific problems from
time to time.

Another point is testing compiler-specific code (i.e. the code that
uses compiler-specific features available since some version). Our
current infrastructure doesn't really suit for that kind of testing,
but still it is possible to see that the code compiles and runs. In
some cases it is possible to verify that it doesn't generate warnings.

If you want to reduce the number of configs to test, you can probably
have a look which compiler versions are used in different Linux
distros (e.g. CentOS, Fedora, OpenSUSE, Ubuntu, Debian) of the latest
LTS and normal releases. For instance, I know gcc 4.4 was used in
Debian Squeeze, but in Wheezy (the latest Debian release at the
moment) it's 4.7, so if no one else uses 4.4 we can probably drop it.
Make a list of what you'd like to drop and post it here so that people
can see which versions specifically are candidates for removal.

> 2) The new version of my runners (Ubuntu 14.04) doesn't easily support the
> old gcc versions, it would take a lot of effort to get some of the old ones
> running. Clang is even worse...it is much more difficult to get even two
> versions running side by side now.

Is it possible to install different versions in chroot environment?

> Instead of running lots of different versions, I was thinking about running
> various compatibility options of the two main compilers...I've seen some
> other test runners with things like libc++,  c++11, c++14, etc.  My real
> question, would developers be better served by these options than different
> versions? Would they prefer both with longer revisit times between each
> test type? Other thoughts?

Having testbeds for newer language versions would be good, but if I
had to choose between more language versions and more compilers I
would choose the latter. It would be ok for me if there were a limited
set of fast turnaround testbeds (e.g. latest widespread gcc in C++03
and C++11 modes) and the more exotic rest were less frequent, as long
as "less frequent" is reasonable. I'd say, any testbed should cycle at
least once a week or so. With longer refresh periods it becomes more
difficult to fix bugs before the release.