[boost.test] 0 return code with expected failures?
Hi, I have a binary that runs unit tests, 3 tests fail and these tests I have marked as being expected failures after analysis. The binary however produces a non-zero exit code even with these tests marked as expected failures . Is there a way to have the binary return a zero exit code when all the failures are expected and non-zero when some other non-expected failure happens? The test setup uses automatic test registration. Regards, Ahmed
Ahmed Badran <ahmed.badran <at> gmail.com> writes:
Is there a way to have the binary return a zero exit code when all the failures are expected and non-zero when some other non-expected failure happens?
What version of Boost are you using? Is this the case for the latest release? Gennadiy
On 3/18/2010 12:20 AM, Gennadiy Rozental wrote:
Ahmed Badran<ahmed.badran<at> gmail.com> writes:
Is there a way to have the binary return a zero exit code when all the failures are expected and non-zero when some other non-expected failure happens?
What version of Boost are you using? Is this the case for the latest release?
I'm using 1.38, I've also tried with 1.40 and it seems to produce the same result. I will try with 1.42
On 3/18/2010 10:56 AM, Ahmed Badran wrote:
On 3/18/2010 12:20 AM, Gennadiy Rozental wrote:
Ahmed Badran<ahmed.badran<at> gmail.com> writes:
Is there a way to have the binary return a zero exit code when all the failures are expected and non-zero when some other non-expected failure happens?
What version of Boost are you using? Is this the case for the latest release?
I'm using 1.38, I've also tried with 1.40 and it seems to produce the same result. I will try with 1.42
The same issue happens with 1.42
Ahmed Badran <ahmed.badran <at> gmail.com> writes:
On 3/18/2010 10:56 AM, Ahmed Badran wrote:
On 3/18/2010 12:20 AM, Gennadiy Rozental wrote:
Ahmed Badran<ahmed.badran<at> gmail.com> writes:
Is there a way to have the binary return a zero exit code when all the failures are expected and non-zero when some other non-expected failure happens?
What version of Boost are you using? Is this the case for the latest release? The same issue happens with 1.42
Can't reproduce the issue. Please try unit_test_example_08 and let me know if you see non zero result code. Gennadiy
On 3/20/2010 12:32 PM, Gennadiy Rozental wrote:
Ahmed Badran<ahmed.badran<at> gmail.com> writes:
On 3/18/2010 10:56 AM, Ahmed Badran wrote:
On 3/18/2010 12:20 AM, Gennadiy Rozental wrote:
Ahmed Badran<ahmed.badran<at> gmail.com> writes:
Is there a way to have the binary return a zero exit code when all the failures are expected and non-zero when some other non-expected failure happens?
What version of Boost are you using? Is this the case for the latest release? The same issue happens with 1.42
Can't reproduce the issue. Please try unit_test_example_08 and let me know if you see non zero result code.
Ahhhh, finally found the difference, example 8 works as I would expect. Turns out that my test case had a BOOST_REQUIRE rather than a BOOST_CHECK (like example 8), so it turns out that a test case that has a BOOST_REQUIRE and is flagged as a known issues still causes a non-zero exit status which is not what I would expect since the test was flagged as a known issue anyway. Is that the intended behavior? Or would you consider revising this behavior in a future release? I appreciate your help. Regards, Ahmed
Ahmed Badran <ahmed.badran <at> gmail.com> writes:
Ahhhh, finally found the difference, example 8 works as I would expect. Turns out that my test case had a BOOST_REQUIRE rather than a BOOST_CHECK (like example 8), so it turns out that a test case that has a BOOST_REQUIRE and is flagged as a known issues still causes a non-zero exit status which is not what I would expect since the test was flagged as a known issue anyway. Is that the intended behavior? Or would you consider revising this behavior in a future release? I appreciate your help.
You can't "expect" fatal errors. The easiest way to deal with this is to store the result in boolean variable and use BOOST_CHECK in it. The rest of the test can be conditioned on the variable name. Gennadiy
On 4/15/2010 6:53 PM, Gennadiy Rozental wrote:
Ahmed Badran<ahmed.badran<at> gmail.com> writes:
Ahhhh, finally found the difference, example 8 works as I would expect. Turns out that my test case had a BOOST_REQUIRE rather than a BOOST_CHECK (like example 8), so it turns out that a test case that has a BOOST_REQUIRE and is flagged as a known issues still causes a non-zero exit status which is not what I would expect since the test was flagged as a known issue anyway. Is that the intended behavior? Or would you consider revising this behavior in a future release? I appreciate your help.
You can't "expect" fatal errors. The easiest way to deal with this is to store the result in boolean variable and use BOOST_CHECK in it. The rest of the test can be conditioned on the variable name.
I guess you're right, the test really wasn't expecting fatal errors, it was more along the lines of require this precondition (the boost require check) before running the actual test within a single unit test. Then again these tests are now considered acceptable failures, but I do not want to disable them for history and tracking purposes. -Ahmed
participants (2)
-
Ahmed Badran -
Gennadiy Rozental