Subject: [boost] [testing] How not to mark up explicit failures
From: David Abrahams (dave_at_[hidden])
Date: 2008-09-04 21:52:25
https://svn.boost.org/trac/boost/ticket/2258 describes a problem with
explicit failure markup that I suspect is not atypical. I believe we
need to establish some ground rules for markup that ensure
* Boost problems don't pass unnoticed
* there is a uniform standard for interpreting test results across
* "non-Boost problems" and non-problems don't add alarm bells for
people reading test results
You can view the markup in question here:
Aside from the specifics in the ticket, the following problems jump out
at me just from looking at that XML:
* even on compilers that do support nonstandard calling conventions,
failures in these tests will never be flagged as problematic
* even on compilers that don't support nonstandard calling conventions,
these tests will fail and add useless noise to the chart.
I can think of a few general principles and practices that will prevent
* Boost regression tests should test the functionality of code that is
expected to work, not to test the capabilities of the platform or C++
There is a place for capability testing, but it needs to be
segregated from regular regression tests somehow. If/when we move to
using CMake globally, capability tests should take place in its
configure stage. In the meantime, such tests must be kept out of
Boost regression testing.
* Tests that do not apply to a given compiler or platform should
ideally not be run on that compiler.
* If we can't find a way to avoid running the test at all, the test
should be written to trivially succeed on that compiler or platform.
* Test annotation wildcards should be written restrictively so that
they only match the compilers/platforms to which they apply.
This is probably just a start; I'd like to hear other ideas. I intend
that this thread will result in an official Boost policy for test
-- Dave Abrahams BoostPro Computing http://www.boostpro.com
Boost list run by bdawes at acm.org, gregod at cs.rpi.edu, cpdaniel at pacbell.net, john at johnmaddock.co.uk