Re: [boost] [build] Test marked as compile-fail is now reported as a failure.

Subject: Re: [boost] [build] Test marked as compile-fail is now reported as a failure.
From: Vladimir Prus (vladimir_at_[hidden])
Date: 2015-01-12 13:37:42

Hi Adam,

On 01/12/2015 06:58 PM, Adam Wulkiewicz wrote:
>>> see e.g.: geographic_custom_point_with_wrong_units
>>> on the page: http://www.boost.org/development/tests/develop/
>>> developer/geometry.html
>>>
>>
>> Why is it surprising that test A fails while test B does not? Are these
>> exactly identical
>> tests?
>>
>
> They're both expected to fail to compile.
>
>
>>
>>   They seem to be defined the same way in the Jamfile:
>>> https://github.com/boostorg/geometry/blob/develop/index/
>>> test/rtree/Jamfile.v2
>>> https://github.com/boostorg/geometry/blob/develop/test/
>>> point_concept/Jamfile.v2
>>>
>>
>> Could it be that the problem with the actual tested functionality.
>>
>
> AFAIU no because the test is reported as a fail only if it's expected to
> compile/link/run but it fails to compile/link/run. If the situation is the
> opposite (expected to fail but doesn't fail) the test is reported as a pass
> (with dark green background). Otherwise the test should be reported as a
> "regular" pass.

I am fairly sure that dark green vs green depends only on status/explicit-failures-markup.xml,
and not on the type of the test. In the absence of any such markup:

- The regular compile test tries to compile a source file, and is deemed a pass if compilation
   succeeds
- The compile-fail test tries to compile a source file, and is deemed a pass if compilation
   fails

Pass is green, fail is yellow.

> Yes, the cause may be located elsewhere, in Geometry, Regression, Test, etc.
>
> For a clean run I'm now unable to build the tests due to some undefined
> references but the problem is most likely located elsewhere.

I think you first need to determine whether the actual compilation of the source file results in
success (i.e. object file is produced) or failure (i.e. errors from compiler). For a compile-fail
test, the compilation is expected to result in failure. If it does not, it's a problem in test itself
or geometry library. If compilation results in failure, but regression matrix shows it as failed test,
then something is wrong down the pipeline.

-- 
Vladimir Prus
CodeSourcery / Mentor Embedded
http://vladimirprus.com