Re: [boost] [contract] help marking regression test expected failures

Subject: Re: [boost] [contract] help marking regression test expected failures
From: Lorenzo Caminiti (lorcaminiti_at_[hidden])
Date: 2017-10-26 02:36:26

On Wed, Oct 25, 2017 at 1:26 PM, Edward Diener via Boost
<boost_at_[hidden]> wrote:
> On 10/25/2017 4:21 PM, Lorenzo Caminiti via Boost wrote:
>>
>> On Wed, Oct 25, 2017 at 8:07 AM, Joaquin M López Muñoz via Boost
>> <boost_at_[hidden]> wrote:
>>>
>>> El 25/10/2017 a las 16:56, Lorenzo Caminiti via Boost escribió:
>>>>
>>>> On Wed, Oct 25, 2017 at 6:38 AM, Edward Diener via Boost
>>>> <boost_at_[hidden]> wrote:
>>>>>
>>>>> On 10/25/2017 3:09 AM, Joaquin M López Muñoz via Boost wrote:
>>>>>>
>>>>>> El 25/10/2017 a las 1:16, Lorenzo Caminiti via Boost escribió:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> https://github.com/boostorg/boost/blob/develop/status/explicit-failures-markup.xml
>>>>>>>
>>>>>>> Doesn't math the expected failures I see on the regression report:
>>>>>>>
>>>>>>> http://www.boost.org/development/tests/develop/developer/contract.html
>>>>>>
>>>>>>
>>>>>> Your changes haven been added to develop but not merged to the release
>>>>>> version of
>>>>>> explicit-failures-markup.xml yet.
>>>>>
>>>>>
>>>>> Why would not an explicit-failures-markup.xml for 'develop' be applied
>>>>> to
>>>>> 'develop' regression testing ?
>>>>
>>>>
>>>> Correct. I am only looking at develop regression testing so the fact
>>>> that explicit-failures-markup.xml is not merged to master/release is
>>>> not the problem.
>>>
>>>
>>> I *think* the xml version from master is used regardless of whether the
>>> reports corresponds
>>> to develop or master.
>>
>>
>> You are right! I just checked and the expected failures showing up on
>> develop regression test report match 100% the markups of
>> explicit-failures-markup.xml from *master* (instead of using the
>> markups from develop).
>>
>> Who can fix this? Making sure that explicit-failures-markup.xml from
>> develop (not master) is used to markup the develop's regression tests.
>
>
> The more general fix is to have the 'develop' regression tests use 'develop'
> everywhere and have the 'master' regression tests use 'master' everywhere.

Agreed.

Someone suggested to me that what maybe needs to be done is here:

https://github.com/boostorg/regression/blob/ci/test-reports/ci_build_results_all.sh#L195

add the line

(cd ${boost} && git checkout origin/${1} --
status/explicit-failures-markup.xml)

but for that to work, here

https://github.com/boostorg/regression/blob/ci/test-reports/ci_build_results_all.sh#L56

--depth 1 must be removed as the above doesn't work on a shallow clone.

However, neither who suggested these changes to me neither I feel
confident doing these changes because we've never touched the testing
infrastructure...

Anyone that can check if the changes above are correct and will make
develop regression test reports use the develop branch instead of
master?

Thanks.
--Lorenzo