Re: CI Testing and failures

From: Andrey Semashev (andrey.semashev_at_[hidden])
Date: 2020-04-28 10:47:39

On 2020-04-27 19:52, Edward Diener via Boost wrote:
> On 4/26/2020 12:07 PM, Andrey Semashev via Boost wrote:
>> On 2020-04-26 18:17, Edward Diener via Boost wrote:
>>> Sometimes CI testing fails because of a bug in a compiler. It is 
>>> possible to designate in a jam file that a particular test should not 
>>> be done for a particular compiler version so that CI testing 
>>> succeeds, and therefore a PR can be more easily merged because the CI 
>>> tests "succeed". But then the fact that the compiler has a bug is 
>>> masked by such a change to the jam file. Is there a solution to such 
>>> a problem, or maybe this is really not a problem at all ? What do 
>>> others think about this ?
>>
>> If the bug affects the library, and you still intend to support it, it 
>> follows that you have to add a workaround for the bug in the library. 
>> The failing test indicates that users of this compiler have an actual 
>> problem.
> 
> So if there is no workaround, should the feature for the particular 
> compiler/version be disabled ? Should the test "succeed" for the feature 
> for the particular compiler/version by either being disabled in Boost 
> Build or by "passing" because the feature is disabled for the 
> compiler/version ?
> 
>>
>> If you don't intend to support this compile, just remove it from the 
>> CI. In the old build matrix this can't be done, except to disable it 
>> in the test/Jamfile.
> 
> I do not think the entire compiler/version should be removed from 
> testing if just a very small part of the library does not work because 
> of a bug. What you are saying is that the tests are meant to succeed 
> rather than show a valid bug in some compiler/version. The opposite 
> viewpoint is that a failing test is a good indication of a 
> compiler/version bug and therefore a failing test is not an indication 
> of a failing library. For the purposes of Github changes, particularly 
> PRs, with CI tests, it does seem as if the tests should succeed, which 
> would back your viewpoint.

That is, of course, for you to decide. If only a small part of the 
library is not supported on the particular compiler while the rest is 
still usable, you can document this limitation and exclude the tests of 
that part from the CI (e.g. via preprocessor checks or by factoring out 
the test to a separate executable and disabling it in the Jamfile).

My main point is that if you want to deal with the compiler bug, you 
should find a workaround and keep the tests running. Otherwise, there's 
no point in running tests for the unsupported code on that compiler.