Re: [Boost-gil] Multi-compiler testing on CircleCI (and channel bug)

Subject: Re: [Boost-gil] Multi-compiler testing on CircleCI (and channel bug)
From: Stefan Seefeld (stefan_at_[hidden])
Date: 2018-04-06 12:30:59

On 06.04.2018 05:18, Mateusz Loskot wrote:
> Hi,
>
> I decided to finally add the CircleCI to our palette of CI-s.
> There, I've configured builds with *all* GCC and clang
> versions I managed to find reliable Docker images [1] for.
>
> That is:
>
> GCC 4.7, 4.8, 4.9
> GCC 5.1, 5.2, 5.3, 5.4, 5.5
> GCC 6.1, 6.2, 6.3, 6.4
> GCC 7.1, 7.2, 7.3
> clang 3.9, 4.0, 5.0
>
> All compile with
> -std=c++11
> variant=debug
> variant=release (ie. <optimization>speed)
>
> (For now, only core tests are enabled, extension tests are disabled.)
>
> Although it may seem unnecessary or superfluous, I decided
> to test wide range of compilers after I noticed some versions reproduce
> the channel [2] tests are failing for some versions and I couldn't see
> the pattern.
> So, I suggest we keep testing against range of GCC/clang versions,
> at least until all CI the builds become perfectly and constantly green :)

Fair enough. But once we get there, I think we should split the test
configs in "essential" ones that really are to be performed on each
check-in, and everything else that we may want to trigger manually. (If
we were using BuildBot for this, I'd suggest some clever tricks to
reduce overhead, such as first running a single test config, and only if
that passes run everything else. Having ~20 test runs perform in
parallel only to then notice a single stupid typo is really wasteful.)

> To summary, from tests against 15+ compilers, the channel failures
> occur only for just two:
>
> - CircleCI with clang 5.0.1 only (see red build job in [3] workflow)
> - Travis CI with GCC 5.4.1 (see [2])
> - my local Linux with GCC 5.4.0 and clang 5.0 (see comments and screens in [2])
>
> Question is, are we indeed suffering from an extremely capricious UB
> or are we hitting some sort of compiler deficiency.

At this point I'm strongly leaning towards the former. It's in the
nature of UB issues that they manifest themselves in funny failure
patterns especially when different levels of optimization are involved.
The many warnings the code still produces (especially those about type
castings and type aliasing) may be our best lead for the time being.

> Just to remind, theres is also the checksum [4] tests failures
> which I can only reproduce with VS2017 x64
>
>
> [1] https://github.com/boostorg/gil/blob/develop/.circleci/config.yml#L15-L74
> [2] https://github.com/boostorg/gil/issues/51
> [3] https://circleci.com/workflow-run/aa52b030-48fa-42eb-bf0e-473b5ee7d96e
> [4] https://github.com/boostorg/gil/issues/49
>
> Best regards,

Thanks for working on those issues !

Stefan

-- 
      ...ich hab' noch einen Koffer in Berlin...