Re: [boost] Release Tools Analysis Part II: Developer Feedback Systems

From: Stefan Seefeld (seefeld_at_[hidden])
Date: 2007-08-17 13:32:29

David Abrahams wrote:
> on Fri Aug 17 2007, Stefan Seefeld <seefeld-AT-sympatico.ca> wrote:
> 
>>> Would QMTest be used to drive multi-host testing across the internet
>>> (i.e. at different testers' sites), or more likely just within local
>>> networks?  If the former, how do its facilities for that compare with
>>> BuildBot?
>> QMTest would typically be used to drive individual 'test runs',
>> presumably only over local networks, 
> 
> Why presumably?  Is there a limitation that prevents it from going out
> to the web?

No. I'm just speculating what users might do with it.

>> and can then be used during the aggregation of the results of such
>> test runs into test reports.
>>
>> As such, it is complementary to the facilities offered by buildbot.
> 
> Can you explain why it makes sense to use two systems?

I'm not quite sure I understand the question. Automating builds
(scheduling build processes triggered by some events) is quite
different from managing test databases.

>>>> Another important point is scalability: While some test suites are
>>>> simple and small, we also deal with test suites that hold many
>>>> thousands of tests (QMTest is used for some of the GCC test suites,
>>>> for example). A test can mean to run a single (local) executable, or
>>>> require a compilation, an upload of the resulting executable to a
>>>> target board 
>>> Target board?
>> Yes (please note that 'target' here is not the same term used above).
>> In the context here it refers to cross-compilation and cross-testing.
> 
> But what is it?

It is some piece of hardware that the test code may need to be uploaded
to in order to be run. QMTest contains logic to do that, if requested
(for example when testing that cross-compiled code runs correctly on
a host platform that is different from the build platform, such as
embedded chips.)

>>>> How could this be useful for boost ?
>>> A good question, but I'm more interested in "how Boost might use it."
>>> That is, something like, "We'd set up a server with a test database.
>>> QMTest would run on the server and drive testing on each testers'
>>> machines, ..." etc.
> 
> Still looking for that.

Yes, I realize that. But as I indicated earlier, I'm not convinced
QMTest is a good tool to schedule / drive that. I'd use a buildbot
setup for that. (Of course you may argue that it is hard to convince
potential testers to install yet another piece of software, but that's
a different argument, I think.)

>>>> I believe the hardest part is the connection between QMTest and
>>>> boost.build. Since boost.build doesn't provide the level of
>>>> introspection QMTest promises, a custom 'boost.build test database'
>>>> implementation needs some special hooks from the build system. I
>>>> discussed that quite a bit with Vladimir.
>>> And what came of it?
>> I'm not sure. boost.build would need to be extended to allow
>> QMTest to gain access to the database structure (the database
>> already exists, conceptually, in terms of the directory layout...).
>> Volodya ?
> 
> There's no a priori reason that Boost.Build needs to maintain the test
> database, is there?

No.

Regards,
                Stefan

-- 
      ...ich hab' noch einen Koffer in Berlin...