Subject: Re: [boost] Linux regression test runners - What do devs want?
From: Niall Douglas (s_sourceforge_at_[hidden])
Date: 2015-01-04 01:19:13
On 3 Jan 2015 at 10:36, Tom Kent wrote:
> I can try to set things up like this again, but there are two reasons I was
> thinking about changing:
> 1) It seems that a lot of the gcc/clang versions have the same pass/fail
> results as the other versions of that compiler, so we're not adding a ton
> (though we may be adding the crucial points where a compiler change breaks
> something...is this something that devs have seen in practice?)
For my automated testing of parts of Boost I only bother testing
builds with multiple compiler versions, and I don't bother with the
unit testing which is generally only done with one clang (thread and
UB sanitiser) and one gcc (right now usually 4.8).
The build testing across compilers is very valuable, and you'd be
amazed how often some random version breaks e.g. GCC 4.7 only. I've
only ever seen unit testing break once on a single compiler only,
that was on GCC 4.8.0. Obviously different code bases may experience
> 2) The new version of my runners (Ubuntu 14.04) doesn't easily support the
> old gcc versions, it would take a lot of effort to get some of the old ones
> running. Clang is even worse...it is much more difficult to get even two
> versions running side by side now.
That's because the LLVM maintainer of the Ubuntu repos only added
multi-version support from 3.5 onwards or so due to a feature request
from me. To get earlier versions of clang which coexist happily on
the same Ubuntu 14.04, look into the hATrayflood repo on launchpad.
He's hacked clangs right back to 2.9 through to 3.6, and they all
I also have GCCs back to 4.6 all coexisting, I think that's from the
Ubuntu toolchain repo on launchpad. I don't need earlier GCCs, so
can't say much about those.
> Instead of running lots of different versions, I was thinking about running
> various compatibility options of the two main compilers...I've seen some
> other test runners with things like libc++, c++11, c++14, etc. My real
> question, would developers be better served by these options than different
> versions? Would they prefer both with longer revisit times between each
> test type? Other thoughts?
For Boost.Thread you can see the build dashboard at
https://github.com/boostorg/thread/. As you'll see, we test 03, 11
and 14 as we have been surprised by random breakage.
Instead of testing libc++ on Linux, I'd suggest a FreeBSD 10 runner
which is also surprisingly good at catching problems on OS X. I've
found you can simply use the FreeBSD 10 default compiler, though note
that the auto detection logic in bootstrap.sh is broken on FreeBSD
and has been for some time (force the toolset to clang, it'll work).
-- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/
Boost list run by bdawes at acm.org, gregod at cs.rpi.edu, cpdaniel at pacbell.net, john at johnmaddock.co.uk