Handling testcases that are known to fail?

From: Douglas Gregor (gregod_at_[hidden])
Date: 2002-01-23 19:05:29

Hello,
  How should we handle testcase that aren't intended to fail, but we know 
they are going to fail? For instance, 
libs/regex/test/c_compiler_checks/wide_posix_api_check.cpp is, of course, 
only going to work if we have wide character support on the system. So when 
BOOST_NO_CWCHAR is defined, there are a few options for handling this 
situation:
  1) Perform the test as-is, and deal with the fact that the failure will 
always be there
  2) Modify the test so that it merely prints out an error string describing 
that wide characters aren't supported, but still passes (because it is a 
known limitation).
  3) Don't run the test at all for the platform. This would require the 
regression testing system to know when a test will always fail, i.e., it 
would need much of the knowledge of the config system.

Personally, #3 would be my favorite if not for the technical hurdles. #1 is 
abominable to me, because we will _never_ achieve a clean build of Boost if 
we allow errors to show up on the regression testing dashboard: it's a 
slippery slope. #2 wins by default in my mind.

        Doug