[Boost-bugs] [Boost C++ Libraries] #1560: Performance testing with boost::test

Subject: [Boost-bugs] [Boost C++ Libraries] #1560: Performance testing with boost::test
From: Boost C++ Libraries (noreply_at_[hidden])
Date: 2008-01-07 22:06:37

#1560: Performance testing with boost::test
---------------------------------------------+------------------------------
 Reporter:  John Pavel <jrp_at_[hidden]>  |       Owner:  rogeeff     
     Type:  Bugs                             |      Status:  new         
Milestone:  To Be Determined                 |   Component:  test        
  Version:  Boost Development Trunk          |    Severity:  Optimization
 Keywords:                                   |  
---------------------------------------------+------------------------------
 I have been trying to use boost::test to do some performance testing.

 With the following macros:

 {{{
 // The basic idea of "test" is to call the function to be timed many
 // times (say up to 1000), throw away the slowest 10% of those times,
 // and average the rest. Why? <shrug> shoot-from-the-hip "theory" and
 // experience that its return is fairly consistent to 2, maybe 3
 // digits. The test function has a "time out" feature where it quits if
 // the accumulated measured time grows beyond 1 second (which may not be
 // appropriate for all tests). But it is easy to get bored when you
 // unexpectedly find yourself waiting 30 minutes for timing result, so
 // that's why it's there. Otoh, I put in a minimum repetition of at
 // least 10, no matter how long the measured time is, so you get at
 // least some accuracy (tweak knobs as you like). Note that the
 // accumulation/averaging happens in double, even though the data and
 // answer are float (just paranoia really). Weakness: If you're timing
 // something that is quicker than the minimum resolution of your timer,
 // this doesn't work. But otherwise, this is better than the
 // traditional loop inside the timer as it throws away those results
 // that happen to get interrupted by your email checker running. :-)

 #include <numeric>
 #include <boost\timer.hpp>

 template <class F>
 double
 time_tests(F f) // f is a function that returns its execution time
 {
         std::vector<double> t;

         // Store time of 10 executions of f
         unsigned int i;
         for (i = 0; i < 10; ++i)
                 t.push_back(f());

         double total_time = std::accumulate(t.begin(), t.end(), 0.0);

         // Keep running until at least 1s of results are available, or
 1000 executions
         while (i < 1000 && total_time < 1.0)
         {
                 t.push_back(f());
                 total_time += t.back();
                 ++i;
         }
         std::sort(t.begin(), t.end());
         t.resize(t.size() * 9 / 10);
         return std::accumulate(t.begin(), t.end(), 0.0) / t.size();
 }


 #define TIMED_TEST_CASE( test_name ) \
         double \
         time_test_##test_name() \
         { \
                 boost::timer t; \
                 { \
                         test_name##_impl();  \
                 } \
                 return t.elapsed(); \
         }

 #define TIMED_AUTO_TEST_CASE( test_name ) \
         void \
         test_name##_impl(); \
         \
         TIMED_TEST_CASE( test_name ) \
         \
         BOOST_AUTO_TEST_CASE( test_name ) \
         { \
                 double execution_time = time_test_##test_name(); \
                 boost::unit_test::unit_test_log.set_threshold_level(
 boost::unit_test::log_messages ); \
                 BOOST_TEST_MESSAGE(BOOST_TEST_STRINGIZE( test_name ).trim(
 "\"" ) << " execution time: " << execution_time << "s"); \
                 BOOST_CHECK( true ); \
         } \
         \
         inline void test_name##_impl()
 }}}

 I can define tests such as

 {{{

 // Boost.Test
 //#define BOOST_AUTO_TEST_MAIN
 // #include <boost/test/unit_test.hpp>
 #define BOOST_TEST_MODULE allTests
 #include <boost/test/unit_test.hpp>

 #include "time_test.h"

 #include <vector>


 #define BaseT float


 BOOST_AUTO_TEST_SUITE(vectors);

 TIMED_AUTO_TEST_CASE( vector_test )
 {
         unsigned int const v1_dim = 6;
         unsigned int const v2_dim = 4;
         unsigned int const v3_dim = 65535;

         std::vector<BaseT> v1(v1_dim, 1.0);
         std::vector< std::vector<BaseT> > v2(v2_dim, v1);
         std::vector< std::vector< std::vector<BaseT> > > v3(v3_dim, v2);


 }

 TIMED_AUTO_TEST_CASE( test2 )
 {
     BOOST_CHECK( true );
 }

 TIMED_AUTO_TEST_CASE( test3 )
 {
         for (int i=0; i<10000; i++)
         {
                 BOOST_CHECK( true );
         }
 }


 BOOST_AUTO_TEST_SUITE_END();

 }}}

 This works, but is not particularly elegant.  Is there a better solution
 (eg, involving defining a class that inherits from one of the existing
 ones)?

-- 
Ticket URL: <http://svn.boost.org/trac/boost/ticket/1560>
Boost C++ Libraries <http://www.boost.org/>
Boost provides free peer-reviewed portable C++ source libraries.