Re: [boost] [test] Really deactivate test

Subject: Re: [boost] [test] Really deactivate test
From: Florian Lindner (mailinglists_at_[hidden])
Date: 2016-12-16 10:31:19

Am 14.12.2016 um 17:25 schrieb Raffi Enficiaud:
> Le 14/12/2016 à 11:20, Florian Lindner a écrit :
>> Am 12.12.2016 um 09:24 schrieb Florian Lindner:
>>> Hello,
>>>
>>> I need to conditionally decativate tests and use a custom decorator for that, like that:
>>>
>>>   virtual void apply(bt::test_unit& tu)
>>>   {
>>>     [...]
>>>     if (std::find(_ranks.begin(), _ranks.end(), rank) == _ranks.end()) {
>>>       std::cout << rank << " Disable, because of rank, test: " << tu.full_name() << std::endl;
>>>       tu.p_default_status.value = bt::test_unit::RS_DISABLED;
>>>       return;
>>>     }
>>>   }
>>>
>>> (i'm happy to provide full source code, if anyone is interested)
>>> As you can see it's deativated on some MPI ranks only.
>>>
>>> However, if a user uses -t "testname" it is activated again.
>>>
>>> Is there a way to really deactivate the test? Or delete it from the test tree?
>>
>> Seems to be next to impossible:
>>
>> test_suite.remove(tu.p_id); has no effect.
>>
>> bt::framework::deregister_test_unit(&tu); causes a SIGSEGV within boost.
>>
>> tu.p_default_status.value = bt::test_unit::RS_DISABLED; as I said above, only disables if not enabled otherwise.
>>
>> Any ideas anyone to completely disable / delete a test case from within a decorator?

[... code of decorator ...]

> What about, instead of trying to remove a test when some conditions are met, adding it when those conditions are not met?

Hey, thanks for your reply!

> I would rather go for an easy solution that is just declaring the test in the init_ of boost.test:
> 
> http://www.boost.org/doc/libs/1_62_0/libs/test/doc/html/boost_test/adv_scenarios/test_module_init_overview.html
> 
> and then for declaring tests manually:
> http://www.boost.org/doc/libs/1_62_0/libs/test/doc/html/boost_test/tests_organization/test_cases/test_organization_nullary.html#boost_test.tests_organization.test_cases.test_organization_nullary.manual_registration

We have a lot of tests, and there are many which should just run on one rank as well as many that run on multiple ranks,
testing parallelized scenarios. I would REALLY like to go with automatic test registration, instead of declaring so many
tests in the init function.

> The disabled tests features are by design this way: we want some tests disabled by default except on some conditions,
> and we want to be able to override those conditions from the runtime command line.
> If a test *cannot* run because of some conditions (say GPU is of the wrong type), then it should be disabled on this
> condition as you are doing with your class, but also the test itself need to check that (like catching an error because
> of the instance of the GPU is of the wrong type).

I understand your design rationale. Please try to understand mine. The precondition decorator does what I need, except
that is flags the test as failed if unmet. I just require a method to skip / delete / do not execute a test if a certain
runtime condition is met and do NOT flag this test as failed.

> For your case, it can just be a BOOST_TEST_REQUIRE on the right ranking (if needed).

TEST_REQUIRE flags the test as failed. Tests which are only scheduled to run on rank 0 should not fail on all other
ranks. They should just no be called / not existent.

Thanks again, but so far I don't see improvement with your sugggestions, unless I got something wrong.

Best,
Florian