Re: [boost] [test] self-registration facilities

From: Johan Nilsson (r.johan.nilsson_at_[hidden])
Date: 2005-12-16 03:41:22

Gennadiy Rozental wrote:
> "Johan Nilsson" <r.johan.nilsson_at_[hidden]> wrote in message
> news:dnre20$vv1$1_at_sea.gmane.org...
>> Gennadiy Rozental wrote:
>>> Hi,
>>>
>>> This message should serve as a temporary source of information
>>> regarding self registration faculties of Boost.Test. I will include
>>> the complete docs in next update.
>>>

[snip]

>
>> One of my pet peeves about CppUnit (and most of the xUnit
>> frameworks) is that there's usually no controllable way of
>> indicating a fixture setup failure. I'm running hardware dependent
>> tests in a couple of my projects (e.g. serial port communications).
>> I need a _supported_ way of indicating a setup error, leading to no
>> further tests being executed in the same suite. Is this possible
>> using Boost.Test?
>
> If I gather correctly you need one per suite fixture. It's last kind
> of fixture not yet supported and  It's in my to do list for update 3.
> Once it's done you could just throw an exception to indicate init
> failure

That sounds about correct. I'll just have to wait and see.

>
>> My main point is that the framework should be prepared for an
>> exception to be thrown from the fixture setup (in this case, during
>> F's ctor), and report it as an setup failure (including the
>> exceptions' "what()" information). And, for the
>> BOOST_FIXTURE_TEST_SUITE, don't run any tests within the suite when
>> setup fails.
>
> BOOST_FIXTURE_TEST_SUITE doing different job - The fixture is per
> test case.

Even so (suite or not), if a fixture throws from the ctor, the framework
should be prepared for that. I tried the following (1.33.1):

---
namespace {
struct Fixture {
    Fixture() { throw std::runtime_error("oops!"); }
};
}
BOOST_FIXTURE_TEST_CASE(Failure, Fixture)
{}
---
And the output was:
---
unknown location(0): fatal error in "Failure": std::runtime_error: oops!
---
Ideally, I'd like the framework to be able to differentiate between a
Fixture construction failure (i.e. xUnit::TestXxx::setUp) and an error
originating from e.g. BOOST_CHECK_xxx. Perhaps having a
boost::test::setup_failure exception class that one could use for this
purpose?
>
>> [snip]
>>
>>> ----------------------------------------------------
>>> BOOST_FIXTURE_TEST_SUITE( suite_name, F )
>>> ----------------------------------------------------
>>>
>>> Similar to the BOOST_AUTO_TEST_SUITE, but force all the test cases
>>> within this test suite to use struct F as a fixture.
>>>
>>> BOOST_FIXTURE_TEST_SUITE( s, F )
>>>
>>> // this test case use F as a fixture
>>> BOOST_AUTO_TEST_CASE( test1 )
>>> {
>>>    // body here
>>> }
>>>
>>> BOOST_AUTO_TEST_SUITE_END()
>>>
>>
>> Do all the test cases within the BOOST_FIXTURE_TEST_SUITE and
>> BOOST_AUTO_TEST_SUITE_END macros share the same instance of F (I'd
>> personally prefer that)?
>
> No. I think that would defeat the purpose. You could always mimic
> what you want with plain Static instance:
>
> struct F{} f;
>
> TEST_CASE( t1 )
> {
>    f.i ...
> }
>
> TEST_CASE( t2 )
> {
>    f.i ...
> }
And could you do this while using suites / fixtures - or is this part of the
new functionality?
>
>> Also, how about having a BOOST_FIXTURE_TEST_SUITE_END macro for
>> symmetry (simply resolving to BOOST_AUTO_TEST_SUITE_END)?
>
> I don't see a need for this. BOOST_AUTO_TEST_SUITE_END ends any test
> suite.
I understood that (as you can see from my writing above). It's just that
there exists BOOST_AUTO_TEST_SUITE and BOOST_AUTO_TEST_SUITE_END (which is
symmetric and nice), but then there's also BOOST_FIXTURE_TEST_SUITE and it's
not very intuitive to use BOOST_ *AUTO* _TEST_SUITE_END to end that suite.
Perhaps use a plain BOOST_TEST_SUITE_END instead?
>
>> Another question, is there any possibility to implement hierarchical
>> auto test suite registrations, e.g.:
>>
>> <i/o test suite> - "io"
>>    <ip test suite> - "ip"
>>        <http test suite> - "http"
>>    <file test suite> - "file"
>>
>> ... etc ...
>
> Yes. BOOST_AUTO_TEST_SUITE could be put inside one another. Check
> examples in libs/example directory.
I've searched, but couldn't find any examples with nested test suites. I 
only found one example with multiple test suites within the same file. Could 
you give me a pointer?
Even if this is possible, does the current implementation require the suites 
to be defined within the same compilation unit? To be able to scale using 
this approach (I'm talking scaling within the context of maintaining large 
sets of unit tests), this should not be necessary.
>
>> With reference to the test runner, it would be great to be able to
>> specify e.g.:
>>
>> <runner> --run-tests "io"        run all tests from
>> "io","ip","http","file" test suites
>> <runner> --run_tests "io/ip"    run all tests from "ip","http" test
>> suites
>
> "Run test by name" is scheduled for update 3.
That's great - should that work with nested test suites as in my example 
above?
// Johan