Re: [boost] [test] --catch_system_errors questions

From: Gennadiy Rozental (rogeeff_at_[hidden])
Date: 2008-05-05 02:19:53

Juergen Hunold wrote:
> Hi ! 
> 
> I'm using Boost.Test for quite a while and now I wonder if the behavior  
> of "--catch_system_errors" is correct.

[...]

> When looking at the output of trunk "with_catch.test", I see
> 
> --snip---
> with_catch: assert_test.cpp:13: void test_assert::test_method(): 
> Assertion `false' failed.
> <TestLog><TestSuite name="Master Test Suite"><TestCase 
> name="test_assert"><Message file="assert_test.cpp" 
> line="11">one</Message><Info file="assert_test.cpp" line="12">check 
> true 
> passed</Info><TestingTime>0</TestingTime></TestCase></TestSuite></TestLog><TestResult><TestSuite 
> name="Master Test Suite" result="passed" assertions_passed="1" 
> assertions_failed="0" expected_failures="0" test_cases_passed="1" 
> test_cases_failed="0" test_cases_skipped="0" 
> test_cases_aborted="0"></TestSuite></TestResult>
> EXIT STATUS: 0
> --snap--

I've copied the content of your test into unit_test_example_01.cpp. My 
local run with msvc produces different output (I've added 2>err.xml 
1>out.xml).

out.xml:
<TestLog><TestSuite name="Master Test Suite"><TestCase 
name="test_assert"><Message file="unit_test_example_01.cpp" 
line="11">one</Message><Info file="unit_test_example_01.cpp" 
line="12">check true passed</Info><Exception name="test_assert">

This application has requested the Runtime to terminate it in an unusual 
way.
Please contact the application&apos;s support team for more information.

<LastCheckpoint file="unit_test_example_01.cpp" 
line="12"></LastCheckpoint></Exception><TestingTime>0</TestingTime></TestCase></TestSuite></TestLog>

err.xml
Assertion failed: false, file unit_test_example_01.cpp, line 13
<TestResult><TestSuite name="Master Test Suite" result="failed" 
assertions_passed="1" assertions_failed="1" expected_failures="0" 
test_cases_passed="0" test_cases_failed="1" test_cases_skipped="0" 
test_cases_aborted="1"></TestSuite></TestResult>

> 
> So, some observations: 
> Obviously, the UTF succeds in catching the "assert".
> The resulting message is not in the XML-Log 

It is. You need to split stdout from stderr. Leading message in stderr 
is unfortunate. I don't believe I can fix it easily

> The return code is 0 (Success).

Not in my expirience.

> A quick look at the new docs prototype at 
> 
> http://www.patmedia.net/~rogeeff/html/utf/usage-recommendations/command-line-specific.html
> 
> found
> "If you got fatal exception somewhere within test case, make program 
> generate coredump by adding extra command line argument "
> 
> This was not easy to spot since the real test cases succeeded with error 
> code 0 and the message was hidden in the log file.

We need to figure out why you have zero result code first.

> I found this because msvc on windows fires up an annoying debug window 
> instead. So it seems this is not working there. 
> 
> So, my questions/suggestions are:
> 
> Is this a bug ? Or a feature ? And if its a feature, where can I find 
> the rationale ?
> 
> I think the caught assertions should lead to a non-zero exit code and  a 
> failing test suite.
> 
> And it would be nice if they got properly XML-formatted (but after a 
> quick scan through the source, this seems difficult...)

It seems to be well formed for the most part.

Gennadiy