Re: [boost] [build] Test marked as compile-fail is now reported as a failure.

Subject: Re: [boost] [build] Test marked as compile-fail is now reported as a failure.
From: Adam Wulkiewicz (adam.wulkiewicz_at_[hidden])
Date: 2015-01-12 19:36:39

Hi Vladimir,

Vladimir Prus wrote:
> I am fairly sure that dark green vs green depends only on 
> status/explicit-failures-markup.xml,
> and not on the type of the test. In the absence of any such markup:
>
> - The regular compile test tries to compile a source file, and is 
> deemed a pass if compilation
>   succeeds
> - The compile-fail test tries to compile a source file, and is deemed 
> a pass if compilation
>   fails
>
> Pass is green, fail is yellow.

Still AFAIU compile-fail should work as expected even if the test wasn't 
marked as failing in the xml file.

>
>> Yes, the cause may be located elsewhere, in Geometry, Regression, 
>> Test, etc.
>>
>> For a clean run I'm now unable to build the tests due to some undefined
>> references but the problem is most likely located elsewhere.
>
> I think you first need to determine whether the actual compilation of 
> the source file results in
> success (i.e. object file is produced) or failure (i.e. errors from 
> compiler). For a compile-fail
> test, the compilation is expected to result in failure. If it does 
> not, it's a problem in test itself
> or geometry library. If compilation results in failure, but regression 
> matrix shows it as failed test,
> then something is wrong down the pipeline.
>

Yes, you're right. The test is compiling, and the object file is 
created. But to detect this I was forced to compile it manually.
The message produced by b2 is slightly missleading. On my machine it 
looks like this:

gcc.compile.c++ 
..\..\bin.v2\libs\geometry\index\test\rtree\rtree_values_invalid.test\gcc-mingw-4.9.1\debug\rtree_values_invalid.o

"g++"  -ftemplate-depth-128 -O0 -fno-inline -Wall -g -std=c++11 
-DBOOST_ALL_NO_LIB=1 -DBOOST_CHRONO_DYN_LINK=1 
-DBOOST_CHRONO_THREAD_DISABLED -DBOOST_SYSTEM_DYN_LINK=1 
-DBOOST_SYSTEM_NO_DEPRECATED -DBOOST_TIMER_DYN_LINK=1  -I"..\.." 
-I"..\..\boost\geometry\extensions\contrib\ttmath" -I"index\test" 
-I"test" -c -o 
"..\..\bin.v2\libs\geometry\index\test\rtree\rtree_values_invalid.test\gcc-mingw-4.9.1\debug\rtree_values_invalid.o" 
"index\test\rtree\rtree_values_invalid.cpp"

...failed gcc.compile.c++ 
..\..\bin.v2\libs\geometry\index\test\rtree\rtree_values_invalid.test\gcc-mingw-4.9.1\debug\rtree_values_invalid.o...
...removing 
..\..\bin.v2\libs\geometry\index\test\rtree\rtree_values_invalid.test\gcc-mingw-4.9.1\debug\rtree_values_invalid.o

So the test fails to compile. The compiler output is not printed, but 
not in all cases. The last line doesn't really mean anything, it might 
as well be some cleanup just in case.

If the test fails to compile (failure not expected) the compiler output 
is printed, bjam's output is nearly the same. Just the last line 
"...removing ..." is not displayed.

On the other hand when the compilation failure is expected and the test 
really fails to compile bjam displays nice (failed-as-expected) instead 
of "...failed gcc.compile.c++".

Would it be possible to display slightly different message when a test 
expected to fail didn't fail, e.g. (passed-unexpectedly), 
(unexpected-pass) or something similar?
At least not "...failed gcc.compile.c++" because the compilation didn't 
really fail.

Regards,
Adam