Re: [Boost-users] [boost.test] Don't report skipped test as failure

Subject: Re: [Boost-users] [boost.test] Don't report skipped test as failure
From: Florian Lindner (mailinglists_at_[hidden])
Date: 2016-07-11 07:50:59

Am 11.07.2016 um 00:43 schrieb Gennadiy Rozental:
> Florian Lindner <mailinglists <at> xgm.de> writes:
> 
>>
>> Hello,
>>
>> I use boost.test precondition in an MPI application to disable test at 
>> ranks != 0 if they do not use MPI:
>>
>> boost::test_tools::assertion_result isMaster(boost::unit_test::test_unit_id)
>> {
>>   return precice::utils::Parallel::getProcessRank() == 0;
>> }

Hello,
thanks for your reply!
 
> Is this compile time condition? If yes you can use enable_if decorator.
> Is this identified by command line parameters? If yes you can use 
> enabled/disabled decorator and switch on the test cases using --run 
> parameter.

It is a true run time condition.
Background:
This test is part of an application that works with MPI. MPI launches several processes in parallel and put them in common communication domain.
Some tests use MPI, some do not and they should be executed only on one rank (rank == 0). Therefore it is a runtime condition, but constant after a process is launched.

Therefore it is deterministic, because there are always ranks {0,1,2,3} and some tests should only be ran in the process where rank == 0.

> If this is actual runtime condition, which depends on something outside of
> unit test, this is not a good unit test design IMO, since it make unit test 
> behavior non deterministic. If you really want to do something like this, you 
> can define global fixture and enable/disable test units manually. There is no 
> strait forward public API to access test unit by name like "a.b.c", but you can
> easily do it yourself using framework::master_test_suite, framework::get and 
> test_suite::get interfaces.

Ok, that adds quite some work. It's not just a handful of tests that should only by ran on the rank 0. Maybe I can introduce another decoration that works just as precondition but does not raise an error. Can the decorators be queried from public API?
> Each test unit has public property p_run_status, which you can update to 
> enable/disable test unit execution.
>>
>> BOOST_AUTO_TEST_SUITE(NearestNeighborMapping)
>>
>> BOOST_AUTO_TEST_CASE(ConsistentNonIncremental,
>>                      * boost::unit_test::precondition(isMaster))
>> {...}
>>
>> That works perfect, I just don't like that sipped test are reported as errors. 
>> Example with all tests deactivated:
> 
> I agree the documentation is somewhat misleading in presenting precondition 
> decorator along with enable/disable decorators. Former is similar to failed 
> dependency and is treated as an error if failed. Later is setup time test tree 
> organization tool.

Thanks for that explanation, however unsatisfying.

Best Regards,
Florian