Boost logo

Boost :

Subject: [boost] [testing] How not to mark up explicit failures
From: David Abrahams (dave_at_[hidden])
Date: 2008-09-04 21:52:25 describes a problem with
explicit failure markup that I suspect is not atypical. I believe we
need to establish some ground rules for markup that ensure

 * Boost problems don't pass unnoticed

 * there is a uniform standard for interpreting test results across

 * "non-Boost problems" and non-problems don't add alarm bells for
   people reading test results

You can view the markup in question here:

Aside from the specifics in the ticket, the following problems jump out
at me just from looking at that XML:

 * even on compilers that do support nonstandard calling conventions,
   failures in these tests will never be flagged as problematic

 * even on compilers that don't support nonstandard calling conventions,
   these tests will fail and add useless noise to the chart.

I can think of a few general principles and practices that will prevent
such problems:

 * Boost regression tests should test the functionality of code that is
   expected to work, not to test the capabilities of the platform or C++

   There is a place for capability testing, but it needs to be
   segregated from regular regression tests somehow. If/when we move to
   using CMake globally, capability tests should take place in its
   configure stage. In the meantime, such tests must be kept out of
   Boost regression testing.

 * Tests that do not apply to a given compiler or platform should
   ideally not be run on that compiler.

 * If we can't find a way to avoid running the test at all, the test
   should be written to trivially succeed on that compiler or platform.

 * Test annotation wildcards should be written restrictively so that
   they only match the compilers/platforms to which they apply.

This is probably just a start; I'd like to hear other ideas. I intend
that this thread will result in an official Boost policy for test


Dave Abrahams
BoostPro Computing

Boost list run by bdawes at, gregod at, cpdaniel at, john at