Re: [boost] Linux regression test runners - What do devs want?

Subject: Re: [boost] Linux regression test runners - What do devs want?
From: Niall Douglas (s_sourceforge_at_[hidden])
Date: 2015-01-04 01:19:13

On 3 Jan 2015 at 10:36, Tom Kent wrote:

> I can try to set things up like this again, but there are two reasons I was
> thinking about changing:
> 1) It seems that a lot of the gcc/clang versions have the same pass/fail
> results as the other versions of that compiler, so we're not adding a ton
> (though we may be adding the crucial points where a compiler change breaks
> something...is this something that devs have seen in practice?)

For my automated testing of parts of Boost I only bother testing 
builds with multiple compiler versions, and I don't bother with the 
unit testing which is generally only done with one clang (thread and 
UB sanitiser) and one gcc (right now usually 4.8).

The build testing across compilers is very valuable, and you'd be 
amazed how often some random version breaks e.g. GCC 4.7 only. I've 
only ever seen unit testing break once on a single compiler only, 
that was on GCC 4.8.0. Obviously different code bases may experience 
different things.

> 2) The new version of my runners (Ubuntu 14.04) doesn't easily support the
> old gcc versions, it would take a lot of effort to get some of the old ones
> running. Clang is even worse...it is much more difficult to get even two
> versions running side by side now.

That's because the LLVM maintainer of the Ubuntu repos only added 
multi-version support from 3.5 onwards or so due to a feature request 
from me. To get earlier versions of clang which coexist happily on 
the same Ubuntu 14.04, look into the hATrayflood repo on launchpad. 
He's hacked clangs right back to 2.9 through to 3.6, and they all 
coexist.

I also have GCCs back to 4.6 all coexisting, I think that's from the 
Ubuntu toolchain repo on launchpad. I don't need earlier GCCs, so 
can't say much about those.

> Instead of running lots of different versions, I was thinking about running
> various compatibility options of the two main compilers...I've seen some
> other test runners with things like libc++,  c++11, c++14, etc.  My real
> question, would developers be better served by these options than different
> versions? Would they prefer both with longer revisit times between each
> test type? Other thoughts?

For Boost.Thread you can see the build dashboard at 
https://github.com/boostorg/thread/. As you'll see, we test 03, 11 
and 14 as we have been surprised by random breakage.

Instead of testing libc++ on Linux, I'd suggest a FreeBSD 10 runner 
which is also surprisingly good at catching problems on OS X. I've 
found you can simply use the FreeBSD 10 default compiler, though note 
that the auto detection logic in bootstrap.sh is broken on FreeBSD 
and has been for some time (force the toolset to clang, it'll work).

Niall

-- 
ned Productions Limited Consulting
http://www.nedproductions.biz/ 
http://ie.linkedin.com/in/nialldouglas/