Re: [boost] [build] Test marked as compile-fail is now reported as a failure.

Subject: Re: [boost] [build] Test marked as compile-fail is now reported as a failure.
From: Adam Wulkiewicz (adam.wulkiewicz_at_[hidden])
Date: 2015-01-12 10:58:54

Hi Vladimir

2015-01-12 7:26 GMT+01:00 Vladimir Prus <vladimir_at_[hidden]>:

>
> Hi Adam,
>
> On 01/11/2015 10:38 PM, Adam Wulkiewicz wrote:
>
>> Hi,
>>
>> Since yesterday one test marked as compile-fail is now reported as a
>> failure on the regression tests page.
>>
>> It's the test: rtree_values_invalid
>> on the page: http://www.boost.org/development/tests/develop/
>> developer/geometry-index.html
>>
>> Surprisingly other tests aren't.
>> see e.g.: geographic_custom_point_with_wrong_units
>> on the page: http://www.boost.org/development/tests/develop/
>> developer/geometry.html
>>
>
> Why is it surprising that test A fails while test B does not? Are these
> exactly identical
> tests?
>

They're both expected to fail to compile.

>
>  They seem to be defined the same way in the Jamfile:
>> https://github.com/boostorg/geometry/blob/develop/index/
>> test/rtree/Jamfile.v2
>> https://github.com/boostorg/geometry/blob/develop/test/
>> point_concept/Jamfile.v2
>>
>
> Could it be that the problem with the actual tested functionality.
>

AFAIU no because the test is reported as a fail only if it's expected to
compile/link/run but it fails to compile/link/run. If the situation is the
opposite (expected to fail but doesn't fail) the test is reported as a pass
(with dark green background). Otherwise the test should be reported as a
"regular" pass.

>
>  AFAIS the oldest run when it failed was of the runner:
>> teeks99-03c-Ubuntu12.04-64 on Sat, 10 Jan 2015 02:05:44.
>> On Fri, 09 Jan 2015 18:36:55 (teeks99-08b-win2012R2-64on64) everything
>> was ok.
>>
>> Furthermore when run locally it's also reported as a failure. Some time
>> ago of course "(failed-as-expected)" message was printed instead.
>> Do you have any idea what may be wrong?
>>
>
> Looking at
>
>         http://www.boost.org/development/tests/develop/
> developer/output/BP%20x86_64%20C++11-boost-bin-v2-libs-
> geometry-index-test-rtree- rtree_values_invalid-test-
> clang-linux-3-5~c14_libc++-debug-debug-symbols-off.html
>
> It does look indeed that complication is successfull, while it should fail
> for the test to pass.

If by successful you mean "the compiler output is present" then indeed it
isn't. However it's not always the case, see:

http://www.boost.org/development/tests/develop/developer/output/BP%20x86_64%20C++11-boost-bin-v2-libs-geometry-index-test-rtree-rtree_values_invalid-test-clang-linux-3-6~c14_libc++-debug-debug-symbols-off.html
http://www.boost.org/development/tests/develop/developer/output/trippels_gcc-5-0_c++11-boost-bin-v2-libs-geometry-index-test-rtree-rtree_values_invalid-test-gcc-5-0-0-release-pch-off.html

Though I can't see any error message there, only warnings. But this is not
the first time I see something like this on the regression tests page.
However the message about trimming of the output is displayed only for the
first of the tests mentioned above.

> Locally for me, with gcc, the
> compilation fails, and the test is deemed passing, so I can't see any
> Boost.Build malfunction here.
>

Yes, the cause may be located elsewhere, in Geometry, Regression, Test, etc.

For a clean run I'm now unable to build the tests due to some undefined
references but the problem is most likely located elsewhere.

> Am I missing something?

Or am I?

Regards,
Adam