Re: [boost] [test] Really deactivate test

Subject: Re: [boost] [test] Really deactivate test
From: Raffi Enficiaud (raffi.enficiaud_at_[hidden])
Date: 2016-12-14 11:25:17

Le 14/12/2016 à 11:20, Florian Lindner a écrit :
> Am 12.12.2016 um 09:24 schrieb Florian Lindner:
>> Hello,
>>
>> I need to conditionally decativate tests and use a custom decorator for that, like that:
>>
>>   virtual void apply(bt::test_unit& tu)
>>   {
>>     [...]
>>     if (std::find(_ranks.begin(), _ranks.end(), rank) == _ranks.end()) {
>>       std::cout << rank << " Disable, because of rank, test: " << tu.full_name() << std::endl;
>>       tu.p_default_status.value = bt::test_unit::RS_DISABLED;
>>       return;
>>     }
>>   }
>>
>> (i'm happy to provide full source code, if anyone is interested)
>> As you can see it's deativated on some MPI ranks only.
>>
>> However, if a user uses -t "testname" it is activated again.
>>
>> Is there a way to really deactivate the test? Or delete it from the test tree?
>
> Seems to be next to impossible:
>
> test_suite.remove(tu.p_id); has no effect.
>
> bt::framework::deregister_test_unit(&tu); causes a SIGSEGV within boost.
>
> tu.p_default_status.value = bt::test_unit::RS_DISABLED; as I said above, only disables if not enabled otherwise.
>
> Any ideas anyone to completely disable / delete a test case from within a decorator?
>
> Thanks,
> Florian
>
>
> /// Boost.Test decorator that makes the test run only on specfic ranks
> class OnRanks : public bt::decorator::base
> {
> public:
>
>   explicit OnRanks(const std::vector<int> & ranks) :
>     _ranks(ranks)
>   {}
>
> private:
>
>   virtual void apply(bt::test_unit& tu)
>   {
>     size_t rank = precice::utils::Parallel::getProcessRank();
>     size_t size = precice::utils::Parallel::getCommunicatorSize();
>
>     if (_ranks.size() > size) {
>       auto& test_suite = bt::framework::master_test_suite();
>       // std::cout << "Removed: " << tu.full_name() << " with ID " << tu.p_id
>                 // << " from test suite " << test_suite.full_name() << std::endl;
>       test_suite.remove(tu.p_id);
>       // bt::framework::deregister_test_unit(&tu);
>       // tu.p_default_status.value = bt::test_case::RS_DISABLED;
>       return;
>     }
>
>     if (std::find(_ranks.begin(), _ranks.end(), rank) == _ranks.end()) {
>       auto& test_suite = bt::framework::master_test_suite();
>       // std::cout << "Removed: " << tu.full_name() << " with ID " << tu.p_id
>                 // << " from test suite " << test_suite.full_name() << std::endl;
>       test_suite.remove(tu.p_id);
>       // bt::framework::deregister_test_unit(&tu);
>       // tu.p_default_status.value = bt::test_case::RS_DISABLED;
>       return;
>     }
>   }
>
>   virtual bt::decorator::base_ptr clone() const
>   {
>     return bt::decorator::base_ptr(new OnRanks(_ranks));
>   }
>
>   std::vector<int> _ranks;
>
> };

What about, instead of trying to remove a test when some conditions are 
met, adding it when those conditions are not met?

I would rather go for an easy solution that is just declaring the test 
in the init_ of boost.test:

http://www.boost.org/doc/libs/1_62_0/libs/test/doc/html/boost_test/adv_scenarios/test_module_init_overview.html

and then for declaring tests manually:
http://www.boost.org/doc/libs/1_62_0/libs/test/doc/html/boost_test/tests_organization/test_cases/test_organization_nullary.html#boost_test.tests_organization.test_cases.test_organization_nullary.manual_registration

The disabled tests features are by design this way: we want some tests 
disabled by default except on some conditions, and we want to be able to 
override those conditions from the runtime command line.
If a test *cannot* run because of some conditions (say GPU is of the 
wrong type), then it should be disabled on this condition as you are 
doing with your class, but also the test itself need to check that (like 
catching an error because of the instance of the GPU is of the wrong type).

For your case, it can just be a BOOST_TEST_REQUIRE on the right ranking 
(if needed).

Hope this helps,
Raffi