Re: [boost] Release Tools Analysis Part II: Developer Feedback Systems

From: David Abrahams (dave_at_[hidden])
Date: 2007-08-17 09:46:48

on Tue Aug 14 2007, Stefan Seefeld <seefeld-AT-sympatico.ca> wrote:

> David Abrahams wrote:
>> Can you give a brief summary of what QMTest actually does and how
>> Boost might use it?
>
> QMTest is a testing harness. 

Meaning, a system for running tests and collecting their results?

> Its concepts are captured in python base classes ('Test', 'Suite',
> 'Resource', 'Target', etc.) which then are implemented to capture
> domain-specific details.  (It is straight forward to customize
> QMTest by adding new test classes, for example).

What are Resource and Target?

> QMTest's central concept is that of a 'test database'. A test
> database organizes tests. It lets users introspect tests (test
> types, test arguments, prerequisite resources, previous test
> results, expectations, etc.), as well as run them (everything or
> only specific sub-suites, by means of different 'target'
> implementations 

I don't understand what you mean by "run them *by means of* 'target'
implementations."

> either in serial, or parallel using multi-threading, multiple
> processes, or even multiple hosts).

Would QMTest be used to drive multi-host testing across the internet
(i.e. at different testers' sites), or more likely just within local
networks?  If the former, how do its facilities for that compare with
BuildBot?

> Another important point is scalability: While some test suites are
> simple and small, we also deal with test suites that hold many
> thousands of tests (QMTest is used for some of the GCC test suites,
> for example). A test can mean to run a single (local) executable, or
> require a compilation, an upload of the resulting executable to a
> target board 

Target board?

> with subsequent remote execution, or other even more fancy things.
>
> Test results are written to 'result streams' (which can be
> customized as most of QMTest).  There is a 'report' command that
> merges the results from multiple test runs into a single test report
> (XML), which can then be translated to whatever output medium is
> desired.
>
> How could this be useful for boost ?

A good question, but I'm more interested in "how Boost might use it."
That is, something like, "We'd set up a server with a test database.
QMTest would run on the server and drive testing on each testers'
machines, ..." etc.

> I found that boost's testing harness lacks robustness. 

Our testing system itself seems to be pretty reliable.  I think it's
the reporting system that lacks robustness.

> There is no way to ask seemingly simple questions such as "what
> tests constitute this test suite ?"  or "what revision / date /
> runtime environment etc. does this result correspond to ?", making
> it hard to assess the overall performance / quality of the software.
>
> I believe the hardest part is the connection between QMTest and
> boost.build. Since boost.build doesn't provide the level of
> introspection QMTest promises, a custom 'boost.build test database'
> implementation needs some special hooks from the build system. I
> discussed that quite a bit with Vladimir.

And what came of it?

-- 
Dave Abrahams
Boost Consulting
http://www.boost-consulting.com
The Astoria Seminar ==> http://www.astoriaseminar.com