Re: [boost] Release Tools Analysis Part II: Developer Feedback Systems

From: David Abrahams (dave_at_[hidden])
Date: 2007-08-17 13:21:16

on Fri Aug 17 2007, Stefan Seefeld <seefeld-AT-sympatico.ca> wrote:

>> Would QMTest be used to drive multi-host testing across the internet
>> (i.e. at different testers' sites), or more likely just within local
>> networks?  If the former, how do its facilities for that compare with
>> BuildBot?
>
> QMTest would typically be used to drive individual 'test runs',
> presumably only over local networks, 

Why presumably?  Is there a limitation that prevents it from going out
to the web?

> and can then be used during the aggregation of the results of such
> test runs into test reports.
>
> As such, it is complementary to the facilities offered by buildbot.

Can you explain why it makes sense to use two systems?

>>> Another important point is scalability: While some test suites are
>>> simple and small, we also deal with test suites that hold many
>>> thousands of tests (QMTest is used for some of the GCC test suites,
>>> for example). A test can mean to run a single (local) executable, or
>>> require a compilation, an upload of the resulting executable to a
>>> target board 
>> 
>> Target board?
>
> Yes (please note that 'target' here is not the same term used above).
> In the context here it refers to cross-compilation and cross-testing.

But what is it?

>>> How could this be useful for boost ?
>> 
>> A good question, but I'm more interested in "how Boost might use it."
>> That is, something like, "We'd set up a server with a test database.
>> QMTest would run on the server and drive testing on each testers'
>> machines, ..." etc.

Still looking for that.

>>> I believe the hardest part is the connection between QMTest and
>>> boost.build. Since boost.build doesn't provide the level of
>>> introspection QMTest promises, a custom 'boost.build test database'
>>> implementation needs some special hooks from the build system. I
>>> discussed that quite a bit with Vladimir.
>> 
>> And what came of it?
>
> I'm not sure. boost.build would need to be extended to allow
> QMTest to gain access to the database structure (the database
> already exists, conceptually, in terms of the directory layout...).
> Volodya ?

There's no a priori reason that Boost.Build needs to maintain the test
database, is there?

-- 
Dave Abrahams
Boost Consulting
http://www.boost-consulting.com
The Astoria Seminar ==> http://www.astoriaseminar.com