From: David Abrahams (dave_at_[hidden])
Date: 2007-08-17 09:46:48
on Tue Aug 14 2007, Stefan Seefeld <seefeld-AT-sympatico.ca> wrote:
> David Abrahams wrote:
>> Can you give a brief summary of what QMTest actually does and how
>> Boost might use it?
> QMTest is a testing harness.
Meaning, a system for running tests and collecting their results?
> Its concepts are captured in python base classes ('Test', 'Suite',
> 'Resource', 'Target', etc.) which then are implemented to capture
> domain-specific details. (It is straight forward to customize
> QMTest by adding new test classes, for example).
What are Resource and Target?
> QMTest's central concept is that of a 'test database'. A test
> database organizes tests. It lets users introspect tests (test
> types, test arguments, prerequisite resources, previous test
> results, expectations, etc.), as well as run them (everything or
> only specific sub-suites, by means of different 'target'
I don't understand what you mean by "run them *by means of* 'target'
> either in serial, or parallel using multi-threading, multiple
> processes, or even multiple hosts).
Would QMTest be used to drive multi-host testing across the internet
(i.e. at different testers' sites), or more likely just within local
networks? If the former, how do its facilities for that compare with
> Another important point is scalability: While some test suites are
> simple and small, we also deal with test suites that hold many
> thousands of tests (QMTest is used for some of the GCC test suites,
> for example). A test can mean to run a single (local) executable, or
> require a compilation, an upload of the resulting executable to a
> target board
> with subsequent remote execution, or other even more fancy things.
> Test results are written to 'result streams' (which can be
> customized as most of QMTest). There is a 'report' command that
> merges the results from multiple test runs into a single test report
> (XML), which can then be translated to whatever output medium is
> How could this be useful for boost ?
A good question, but I'm more interested in "how Boost might use it."
That is, something like, "We'd set up a server with a test database.
QMTest would run on the server and drive testing on each testers'
machines, ..." etc.
> I found that boost's testing harness lacks robustness.
Our testing system itself seems to be pretty reliable. I think it's
the reporting system that lacks robustness.
> There is no way to ask seemingly simple questions such as "what
> tests constitute this test suite ?" or "what revision / date /
> runtime environment etc. does this result correspond to ?", making
> it hard to assess the overall performance / quality of the software.
> I believe the hardest part is the connection between QMTest and
> boost.build. Since boost.build doesn't provide the level of
> introspection QMTest promises, a custom 'boost.build test database'
> implementation needs some special hooks from the build system. I
> discussed that quite a bit with Vladimir.
And what came of it?
-- Dave Abrahams Boost Consulting http://www.boost-consulting.com The Astoria Seminar ==> http://www.astoriaseminar.com
Boost list run by bdawes at acm.org, gregod at cs.rpi.edu, cpdaniel at pacbell.net, john at johnmaddock.co.uk