Re: [Boost-cmake] Labels for known failures

Subject: Re: [Boost-cmake] Labels for known failures
From: David Cole (david.cole_at_[hidden])
Date: 2009-05-15 06:47:27

CDash does not have filters on the viewTest page yet, but it is planned for
a future CDash release.

It's cool to see somebody using labels already... Nice job, Doug!

David Cole
Kitware, Inc.

On Fri, May 15, 2009 at 12:54 AM, Doug Gregor <doug.gregor_at_[hidden]> wrote:

> I've added the ability to mark up known failures in the regression
> tests directly in the CMake files for those tests. For example, here's
> one such description of known failures:
>
>  boost_test_run(optional_test_ref
>    KNOWN_FAILURES "gcc-4.[0-3].[0-9]-.*")
>
> The boost_test_run macro use creates the test itself (called
> optional_test_ref), which compilers optional_test_ref.cpp into an
> executable and runs it. The KNOWN_FAILURES keyword argument is
> followed by one or more regular expressions that indicate what
> compiler/platform combinations are known to fail. In this case, we're
> matching GCC 4.0.x-4.3.x on any platform. The compiler/platform
> combinations come from the build names that you see on the dashboard,
> e.g.,
>
>  http://www.cdash.org/CDashPublic/index.php?project=Boost
>
> Sometimes it isn't convenient to add known failures in the
> boost_test_* macro invocation that actually builds the test, so you
> can use the boost_test_known_failures macro to do the same thing,
> e.g.,
>
>  boost_test_known_failures(test_ref_wrapper_tricky_std "gcc-4.0.[0-9]-.*")
>
> When the current build name matches one of the regular expressions
> given, we attach the "known-failure" label to that test. You can see
> that this label shows up in the listing for tests here:
>
>  http://www.cdash.org/CDashPublic/viewTest.php?onlyfailed&buildid=23108
>
> In the future, we'll figure out some way to (optionally) filter out
> any tests with the "known-failure" label when displaying the results
> in CDash, so that we can easily see new/unknown failures on the
> dashboard. CDash might already be able to do this, but I couldn't
> figure out how.
>
> Note that, for these labels to work, the tester needs to be running
> CMake 2.7.x (which is currently in CMake CVS but isn't yet an official
> release). This is pretty important, because labels are also going to
> be used to give the library-centric view of the dashboard.
>
> Comments welcome!
>
>  - Doug
> _______________________________________________
> Boost-cmake mailing list
> Boost-cmake_at_[hidden]
> http://lists.boost.org/mailman/listinfo.cgi/boost-cmake
>