I have been using Boost.test to run some simple unit tests as I build my code. I've come across a case where I need to test if my code runs within an acceptable time limit. I know that the execution monitor has a timeout system for testing that sort of thing, but I can't figure out how it is supposed to be integrated into the unit test framework. Ideally, I'd love to just have a BOOST_CHECK_TIMED( f(), timeout) that I could invoke to ensure that the code returns true, and returns within the specified timeout interval. -- Stirling Westrup Programmer, Entrepreneur. https://www.linkedin.com/e/fpf/77228 http://www.linkedin.com/in/swestrup http://technaut.livejournal.com http://sourceforge.net/users/stirlingwestrup
AMDG On 03/22/2011 11:52 PM, Stirling Westrup wrote:
I have been using Boost.test to run some simple unit tests as I build my code. I've come across a case where I need to test if my code runs within an acceptable time limit. I know that the execution monitor has a timeout system for testing that sort of thing, but I can't figure out how it is supposed to be integrated into the unit test framework.
Well, you can just use boost::execution_monitor. There's nothing incompatible between it and the Unit Test Framework. Unfortunately, as the documentation notes: unit_test::readwrite_property<int> p_timeout; // Specifies the seconds that elapse before a timer_error occurs. May be ignored on some platforms. I tried the following on both Linux and Windows, and it didn't work on Windows. #include <boost/test/included/execution_monitor.hpp> #define BOOST_TEST_MAIN #include <boost/test/included/unit_test.hpp> int f() { for(;;); } BOOST_AUTO_TEST_CASE(test_timeout) { boost::execution_monitor monitor; monitor.p_timeout.set(5); monitor.execute(&f); }
Ideally, I'd love to just have a BOOST_CHECK_TIMED( f(), timeout) that I could invoke to ensure that the code returns true, and returns within the specified timeout interval.
In Christ, Steven Watanabe
On Wed, Mar 23, 2011 at 12:27 PM, Steven Watanabe <watanabesj@gmail.com> wrote:
AMDG
On 03/22/2011 11:52 PM, Stirling Westrup wrote:
I have been using Boost.test to run some simple unit tests as I build my code. I've come across a case where I need to test if my code runs within an acceptable time limit. I know that the execution monitor has a timeout system for testing that sort of thing, but I can't figure out how it is supposed to be integrated into the unit test framework.
Well, you can just use boost::execution_monitor. There's nothing incompatible between it and the Unit Test Framework.
Unfortunately, as the documentation notes:
unit_test::readwrite_property<int> p_timeout; // Specifies the seconds that elapse before a timer_error occurs. May be ignored on some platforms.
I tried the following on both Linux and Windows, and it didn't work on Windows.
#include <boost/test/included/execution_monitor.hpp>
#define BOOST_TEST_MAIN #include <boost/test/included/unit_test.hpp>
int f() { for(;;); }
BOOST_AUTO_TEST_CASE(test_timeout) { boost::execution_monitor monitor; monitor.p_timeout.set(5); monitor.execute(&f); }
Ideally, I'd love to just have a BOOST_CHECK_TIMED( f(), timeout) that I could invoke to ensure that the code returns true, and returns within the specified timeout interval.
Thanks! Your example is simple enough that I can see how to work it into my unit tests. Luckily I'm benchmarking my code under Linux, so I don't have to worry about other platforms when it comes to these tests. -- Stirling Westrup Programmer, Entrepreneur. https://www.linkedin.com/e/fpf/77228 http://www.linkedin.com/in/swestrup http://technaut.livejournal.com http://sourceforge.net/users/stirlingwestrup
On 2011-03-23 07:52, Stirling Westrup wrote:
I have been using Boost.test to run some simple unit tests as I build my code. I've come across a case where I need to test if my code runs within an acceptable time limit. I know that the execution monitor has a timeout system for testing that sort of thing, but I can't figure out how it is supposed to be integrated into the unit test framework.
Ideally, I'd love to just have a BOOST_CHECK_TIMED( f(), timeout) that I could invoke to ensure that the code returns true, and returns within the specified timeout interval.
I just came with a basic solution that I have not fully tested but seems to work. However it demands some optimization. (Please, be soft on me, I have not been working much on this one). I created a ( TestCaseName, TimeOutValue ) macro. This macro is extended to something like #define BOOST_AUTO_TEST_CASE_TIMED( NAME , TIMEOUT) \ class NAME##_timeout { \ public: \ NAME##_timeout(){ConfigurationVisitor::getInstance()->registerTimeOut(#NAME, TIMEOUT);} \ }; \ NAME##_timeout* NAME##_timeInstance = new NAME##_timeout(); \ BOOST_AUTO_TEST_CASE( NAME ) I am using test_tree_visitor implementation - ConfigurationVisitor -instantiated before the boost::monitor execution is called. void ConfigurationVisitor::visit( test_case const& itc ) { test_case & tc=*(test_case *)&itc; // Hack ! tc.p_timeout.set(boost::unit_test::class_property<unsigned>(getTimeOut(tc.p_name.value))); } This will update test_case timeout to the value return by getTimeOut (0 if none, of the actual TimeOutvalue registered for the test case. This test tree visitor also exposed some static functions : static void registerTimeOut(std::string iTestName, int iTimeOut); static int getTimeOut(std::string iTestName); The time out value is store in a std::map. Runtime, before calling the ::boost::unit_test::unit_test_main, I do traverse_test_tree(framework::master_test_suite().p_id, *ConfigurationVisitor::getInstance()); At the end, I got something like that : BOOST_AUTO_TEST_CASE_TIMED( MyTestCase1, 5 ) { while(1){ usleep(1000); BOOST_TEST_MESSAGE("HelloWorld"); } } And log will be : Entering test case "MyTestCase1" HelloWorld HelloWorld HelloWorld HelloWorld HelloWorld unknown location(0): fatal error in "MyTestCase2": signal: SIGALRM (timeout while executing function) .../MyTimeOutTestSuite.cpp(64): last checkpoint Leaving test case "MyTestCase1" ----- However, the hack in the visit function is actually my main concern about this solution. It may not be what you are look for, but I hope this will help though. Guillaume.
On Thu, Mar 24, 2011 at 11:37 AM, Guillaume Theraud <gui@futarque.com> wrote:
On 2011-03-23 07:52, Stirling Westrup wrote:
I have been using Boost.test to run some simple unit tests as I build my code. I've come across a case where I need to test if my code runs within an acceptable time limit. I know that the execution monitor has a timeout system for testing that sort of thing, but I can't figure out how it is supposed to be integrated into the unit test framework.
Ideally, I'd love to just have a BOOST_CHECK_TIMED( f(), timeout) that I could invoke to ensure that the code returns true, and returns within the specified timeout interval.
I just came with a basic solution that I have not fully tested but seems to work. However it demands some optimization. (Please, be soft on me, I have not been working much on this one).
I created a ( TestCaseName, TimeOutValue ) macro. This macro is extended to something like
#define BOOST_AUTO_TEST_CASE_TIMED( NAME , TIMEOUT) \ class NAME##_timeout { \ public: \ NAME##_timeout(){ConfigurationVisitor::getInstance()->registerTimeOut(#NAME, TIMEOUT);} \ }; \ NAME##_timeout* NAME##_timeInstance = new NAME##_timeout(); \ BOOST_AUTO_TEST_CASE( NAME )
I am using test_tree_visitor implementation - ConfigurationVisitor - instantiated before the boost::monitor execution is called.
void ConfigurationVisitor::visit( test_case const& itc ) { test_case & tc=*(test_case *)&itc; // Hack !
tc.p_timeout.set(boost::unit_test::class_property<unsigned>(getTimeOut(tc.p_name.value))); } This will update test_case timeout to the value return by getTimeOut (0 if none, of the actual TimeOutvalue registered for the test case.
This test tree visitor also exposed some static functions :
static void registerTimeOut(std::string iTestName, int iTimeOut); static int getTimeOut(std::string iTestName);
The time out value is store in a std::map.
Runtime, before calling the ::boost::unit_test::unit_test_main, I do traverse_test_tree(framework::master_test_suite().p_id, *ConfigurationVisitor::getInstance());
At the end, I got something like that :
BOOST_AUTO_TEST_CASE_TIMED( MyTestCase1, 5 ) { while(1){ usleep(1000); BOOST_TEST_MESSAGE("HelloWorld"); } }
And log will be :
Entering test case "MyTestCase1" HelloWorld HelloWorld HelloWorld HelloWorld HelloWorld unknown location(0): fatal error in "MyTestCase2": signal: SIGALRM (timeout while executing function) .../MyTimeOutTestSuite.cpp(64): last checkpoint Leaving test case "MyTestCase1"
-----
However, the hack in the visit function is actually my main concern about this solution. It may not be what you are look for, but I hope this will help though.
Guillaume.
Thanks for this response, as it does look like something that I could use, but I have NO IDEA how I am supposed to make use of what you wrote here. You seem to be assuming not only a greater knowledge of the internals of Boost.test than I have, but also a greater knowledge than can be gleamed from the current documentation. -- Stirling Westrup Programmer, Entrepreneur. https://www.linkedin.com/e/fpf/77228 http://www.linkedin.com/in/swestrup http://technaut.livejournal.com http://sourceforge.net/users/stirlingwestrup
participants (3)
-
Guillaume Theraud -
Steven Watanabe -
Stirling Westrup