[Boost-bugs] [Boost C++ Libraries] #7419: Support multiple calls to framework::init() allowing wrappers to support running tests using test tools in full systems

Subject: [Boost-bugs] [Boost C++ Libraries] #7419: Support multiple calls to framework::init() allowing wrappers to support running tests using test tools in full systems
From: Boost C++ Libraries (noreply_at_[hidden])
Date: 2012-09-24 23:56:14

#7419: Support multiple calls to framework::init() allowing wrappers to support
running tests using test tools in full systems
------------------------------------------------+---------------------------
 Reporter:  Jamie Allsop <ja11sop@…>            |       Owner:  rogeeff
     Type:  Bugs                                |      Status:  new    
Milestone:  To Be Determined                    |   Component:  test   
  Version:  Boost Development Trunk             |    Severity:  Problem
 Keywords:                                      |  
------------------------------------------------+---------------------------
 Boost.Test is designed on the assumption that a special `main()` function
 will be provided to ensure the test framework is initialised properly and
 that the function needed to add test suites for execution is properly
 provided. This works well for simple test programs.

 However another important use case we have and I'm sure others too is that
 we also wish to run our production systems 'as-if' in production but have
 tests execute when run in a 'testing' mode.

 This has the benefit of allowing full use of all the Boost.Test tool
 facilities and makes test points easily recognisable. Additionally it also
 means the test out analysis tools that we use for Boost.Test in our unit
 testing can be used without modification. Hence on a coding level we use
 the same test code as for unit tests and we then benefit from being able
 to directly interpret the results, just as we do for normal unit tests.

 In order to support this within the Boost.Test framework (we use the unit-
 test framework for this as we simply require access to the tools and
 reporting facilities) we must make it possible to write a test runner
 wrapper than can ultimately call `boost::unit_test::framework::init()`
 more than once.

 A minimal test runner class might look like this (taken from a real code
 but stripped down so will not work as is):

 {{{
 #!cpp

 namespace test {

 class runner
 {
 public:

     using master_test_suite_t  = boost::unit_test::master_test_suite_t;
     using test_suite_t         = boost::unit_test::test_suite;
     using call_add_tests_t     = boost::function<void(
 master_test_suite_t& )>;

 public:

     static
     void store_properties( /* properties from commandline or config file
 */ )
     {
         auto& Runner = instance();
         Runner.store_arguments( /* properties */ );
     }

     static
     int run( const std::string& Title, const call_add_tests_t& Callback )
     {
         auto& Runner = instance();

         Runner.set_title( Title );
         Runner.set_add_tests_callback( Callback );

 #ifdef BOOST_TEST_ALTERNATIVE_INIT_API
         boost::unit_test::init_unit_test_func init_func =
 &runner::init_unit_test;
 #else
         boost::unit_test::init_unit_test_func init_func =
 &runner::init_unit_test_suite;
 #endif
         return boost::unit_test::unit_test_main( init_func, Runner.argc(),
 Runner.argv() );
     }

 private:

     runner()
     {
     }

     static runner& instance()
     {
         static runner Runner;
         return Runner;
     }

 private:

     static
     bool init_unit_test()
     {
         init_unit_test_suite( 0, 0 );
         return true;
     }

     static
     boost::unit_test::test_suite* init_unit_test_suite( int argc, char*
 argv[] )
     {
         auto AddTests = runner::instance().get_add_tests_callback();
         if( AddTests )
         {
             AddTests( boost::unit_test::framework::master_test_suite() );
         }
         return 0;
     }

 private:

     void store_arguments( /* properties */  )
     {
         ArgumentStrings_ = /* copy of Boost.Test relevant properties */
         Arguments_ = /* pointers to actual strings in ArgumentStrings_ */
     }

 private:

     int argc() const
     {
         return ArgumentStrings_.size();
     }

     char** argv() const
     {
         return const_cast<char**>( &Arguments_[0] );
     }

     void set_title( const std::string& Title )
     {
         if( ArgumentStrings_.empty() )
         {
             ArgumentStrings_.push_back( Title );
             Arguments_.push_back( ArgumentStrings_[0].c_str() );
             Arguments_.push_back( 0 );
         }
         else
         {
             ArgumentStrings_[0] = Title;
             Arguments_[0] = ArgumentStrings_[0].c_str();
         }
     }

     void set_add_tests_callback( const call_add_tests_t& Callback )
     {
         Callback_ = Callback;
     }

     const call_add_tests_t& get_add_tests_callback()
     {
         return Callback_;
     }

 private:

     call_add_tests_t            Callback_;
     std::vector<std::string>    ArgumentStrings_;
     std::vector<const char*>    Arguments_;
 };

 } // test
 }}}

 We would use a runner like this as follows:

 {{{
 #!cpp
     // Implement a function to add tests to the Master Test Suite
     // Note this can be a member function and simply call
     // another member to collate the test suites
     void add_tests( test::runner::master_test_suite_t& MasterSuite )
     {
         MasterSuite.add( scenario_test_suite() );
     }

     // A
     test::runner::test_suite_t* scenario_test_suite()
     {
         auto* test_suite = BOOST_TEST_SUITE( Name_.c_str() );

         test_suite->add( BOOST_TEST_CASE( boost::bind(
 &self_t::validate_incoming_messages, this->shared_from_this() ) ) );
         test_suite->add( BOOST_TEST_CASE( boost::bind(
 &self_t::validate_outgoing_messages, this->shared_from_this() ) ) );
         test_suite->add( BOOST_TEST_CASE( boost::bind(
 &self_t::validate_data_messages,     this->shared_from_this() ) ) );

         return test_suite;
     }

     // Calling test() actually causes the tests to be executed
     // We assume in this example that the object that
     // test() belongs to is a shared_ptr
     void test()
     {
         test::runner::run( Name_, boost::bind( &self_t::add_tests,
 this->shared_from_this(), _1 ) );
     }
 }}}

 The attached patch makes the above code possible by doing 2 things:

   1. If the macro `BOOST_TEST_USE_QUALIFIED_COMMANDLINE_ARGUMENTS` is
 defined then the Boost.Test command line arguments are scoped using
 `boost.test.*` as a prefix. This means commandline arguments used to
 control tests will not interfere with command line arguments used to
 control the code being tested. `--log_level` is an excellent example of
 why you might need to do that. `--boost.test.log_level` is much clearer
 and will not clash.

   2. Calls to `boost::unit_test::framework::init()` will occur each time
 `run()` is called. However with the patch each call resets the framework
 to a consistent initial state and the tests proceed as expected as if the
 framework had been called once from `main()`.

 We are using this patch with boost 1.48 right now (and so it is very
 slightly different due to the changes between 1.48 and trunk) and it
 allows us to run our main processes in a 'test' mode generating full test
 result output with timings that are then sent to our build server. It
 works very well, has no impact on normal unit test usage and is a very
 small change for a very big gain.

 This patch along with the patches attached to these tickets:

 [[TicketQuery(id=7397|7410|7417)]]

 ...combine to significantly improve the value of Boost.Test in terms of
 integration with thirdparty analysis tools and integration with other code
 libraries.

 Given the small changes that are made I'd like to see them hopefully make
 it into the next boost release.

-- 
Ticket URL: <https://svn.boost.org/trac/boost/ticket/7419>
Boost C++ Libraries <http://www.boost.org/>
Boost provides free peer-reviewed portable C++ source libraries.