Re: [boost] Release Tools Analysis Part II: Developer Feedback Systems

From: David Abrahams (dave_at_[hidden])
Date: 2007-08-18 15:45:31

on Fri Aug 17 2007, Stefan Seefeld <seefeld-AT-sympatico.ca> wrote:

> David Abrahams wrote:
>
>> So QMTest doesn't schedule build/test processes?  It's just a database
>> manager?
>
> Yes it does 'schedule' them, but according to rather specific requirements.
> You wouldn't tell QMTest to run a test suite nightly, for example, or
> hook up to some other external triggers (checkins, say). Rather, you
> specify what (sub-) testsuite to run, and QMTest will work out the order
> etc.

The order in which to run tests?  I don't believe we have any
dependency relationships (at least, not encoded ones) that can help
QMTest.

>>>>>>> How could this be useful for boost ?
>>>>>> A good question, but I'm more interested in "how Boost might use it."
>>>>>> That is, something like, "We'd set up a server with a test database.
>>>>>> QMTest would run on the server and drive testing on each testers'
>>>>>> machines, ..." etc.
>>>> Still looking for that.
>>> Yes, I realize that. But as I indicated earlier, I'm not convinced
>>> QMTest is a good tool to schedule / drive that. 
>> 
>> I meant I want some kind of analogous statement about a way we could
>> use it that you *are* convinced of.
>
> * handle all test suite runs through QMTest

too vague.

> * aggregate test results in a central place using QMTest, 

So QMTest stores results, OK.

> and manage interpretation (including expectations) 

How would one do that?

> to generate test reports.

Does QMTest generate reports?

>>>> There's no a priori reason that Boost.Build needs to maintain the test
>>>> database, is there?
>>> No.
>> 
>> So what are the alternatives to that arrangement?
>
> A boost-specific test database implementation could work like this:
>
> * by default, map each source file under libs/*/test/ to a test id,
>   and provide a default test type which
>     1) compiles,
>     2) runs the result,
>     3) checks exit status and output.
>   Default compile and link options could be generated per component
>   (boost library).
>
> * For cases where the above doesn't work, some special test implementation
>   can be used (that incorporate the special rules now part of the various
>   Jamfiles).

I think I understand.  Essentially, one would need to implement Python
classes whose instances represent each test and know how to do the
testing.  One could generate Jamfiles for the difficult cases.  But
how would we represent the tests?  Python code?  An actual database?

As I see it right now, the most significant benefit available from
QMTest is in the fact that it robustly controls the running of each
test, capturing its results, and comparing those results with what's
expected.  Is that right?

-- 
Dave Abrahams
Boost Consulting
http://www.boost-consulting.com
The Astoria Seminar ==> http://www.astoriaseminar.com