|
|
Boost Interest : |
Subject: [Boost-cmake] Labels for known failures
From: Doug Gregor (doug.gregor_at_[hidden])
Date: 2009-05-15 00:54:30
I've added the ability to mark up known failures in the regression
tests directly in the CMake files for those tests. For example, here's
one such description of known failures:
boost_test_run(optional_test_ref
KNOWN_FAILURES "gcc-4.[0-3].[0-9]-.*")
The boost_test_run macro use creates the test itself (called
optional_test_ref), which compilers optional_test_ref.cpp into an
executable and runs it. The KNOWN_FAILURES keyword argument is
followed by one or more regular expressions that indicate what
compiler/platform combinations are known to fail. In this case, we're
matching GCC 4.0.x-4.3.x on any platform. The compiler/platform
combinations come from the build names that you see on the dashboard,
e.g.,
http://www.cdash.org/CDashPublic/index.php?project=Boost
Sometimes it isn't convenient to add known failures in the
boost_test_* macro invocation that actually builds the test, so you
can use the boost_test_known_failures macro to do the same thing,
e.g.,
boost_test_known_failures(test_ref_wrapper_tricky_std "gcc-4.0.[0-9]-.*")
When the current build name matches one of the regular expressions
given, we attach the "known-failure" label to that test. You can see
that this label shows up in the listing for tests here:
http://www.cdash.org/CDashPublic/viewTest.php?onlyfailed&buildid=23108
In the future, we'll figure out some way to (optionally) filter out
any tests with the "known-failure" label when displaying the results
in CDash, so that we can easily see new/unknown failures on the
dashboard. CDash might already be able to do this, but I couldn't
figure out how.
Note that, for these labels to work, the tester needs to be running
CMake 2.7.x (which is currently in CMake CVS but isn't yet an official
release). This is pretty important, because labels are also going to
be used to give the library-centric view of the dashboard.
Comments welcome!
- Doug