Re: [boost] Handling testcases that are known to fail?

From: Jeremy Siek (jsiek_at_[hidden])
Date: 2002-01-23 21:53:53

One fairly common way to handle this sort of this is to have a list of
tests that are expected to fail. This list is subtracted from the list of
normal (real) failures for most display/report purposes. Developers and
maintainers can add and remove tests from the expected fail list. I guess
this is similar to option 3), but a bit simplified because the list of
expected failures is created by hand instead of deduced from the config
system or otherwise.

On Wed, 23 Jan 2002, Douglas Gregor wrote:
gregod> Hello,
gregod>   How should we handle testcase that aren't intended to fail, but we know
gregod> they are going to fail? For instance,
gregod> libs/regex/test/c_compiler_checks/wide_posix_api_check.cpp is, of course,
gregod> only going to work if we have wide character support on the system. So when
gregod> BOOST_NO_CWCHAR is defined, there are a few options for handling this
gregod> situation:
gregod>   1) Perform the test as-is, and deal with the fact that the failure will
gregod> always be there
gregod>   2) Modify the test so that it merely prints out an error string describing
gregod> that wide characters aren't supported, but still passes (because it is a
gregod> known limitation).
gregod>   3) Don't run the test at all for the platform. This would require the
gregod> regression testing system to know when a test will always fail, i.e., it
gregod> would need much of the knowledge of the config system.
gregod>
gregod> Personally, #3 would be my favorite if not for the technical hurdles. #1 is
gregod> abominable to me, because we will _never_ achieve a clean build of Boost if
gregod> we allow errors to show up on the regression testing dashboard: it's a
gregod> slippery slope. #2 wins by default in my mind.
gregod>
gregod> 	Doug
gregod>
gregod> Info: http://www.boost.org  Send unsubscribe requests to: <mailto:boost-unsubscribe_at_[hidden]>
gregod>
gregod> Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/
gregod>
gregod>

----------------------------------------------------------------------
 Jeremy Siek                          http://php.indiana.edu/~jsiek/
 Ph.D. Student, Indiana Univ. B'ton   email: jsiek_at_[hidden]
 C++ Booster (http://www.boost.org)   office phone: [phone]
----------------------------------------------------------------------