Re: [boost] Re: Re: Re: How do regressions tests work?

From: Stefan Seefeld (seefeld_at_[hidden])
Date: 2003-11-19 11:48:50

Vladimir Prus wrote:

> Ah, in other words your propose opposite of what I've thought. Not using
> QMTest to drive bjam invocations, but using bjam to driver qmtest
> invocations. 

precisely.

>>Anyways, my whole point is to show that even though qmtest itself
>>doesn't manage dependencies, it's easy enough to hook it up with
>>tools that are designed to do just that.
> 
> 
> Hm.. this resembles an idea that I once had -- tat regression system should
> have some 'execution monitor' which is told 'compile that file' or 'run
> that program'. It does that, recording interestring info like output, exit
> status, spend time and used memory, or something else.

exactly. And all that is part of the semantics of a particular test
type. So you have to write a test class (in python) which, in it's 'Run'
method does whatever you want to actually test.

> Using QMTest as such execution monitor might be good idea. But sure, there
> are some problems. At least, you need to teach it to use Jamfiles as test
> database. And second, it's not clear how to pass command lines to execute.
> And third, it's not clear what to do if multiple toolsets are to be tests.

Yes, you need to tell qmtest how your tests map to your code. As I 
originally used Makefiles to drive that, I wrote a qmtest database 
(http://www.codesourcery.com/qm/qmtest_downloads/qm-2.0.3/manual.html/sec-qmtest-concepts.html#sec-test-database)
that uses some 'Makefile'-like scripts that are actually python code, 
but which are the user-visible access point, i.e. where he adds tests to 
the database. That's where I would define the environment (build 
variables, say), and the tests (source file and other test specific 
resources needed to run it).

Toolsets nicely map to qmtest's 'context' 
(http://www.codesourcery.com/qm/qmtest_downloads/qm-2.0.3/manual.html/sec-qmtest-concepts.html#sec-context).
Again, what I did (based on make, not bjam) is to have a toplevel 
'Makefile' setting up the database for a specific platform. All 
subsequent 'qmtest run' executions will use those parameters, so you
have a mapping between test database instance and toolset.

> It might be still possible to reuse the test running/result storing
> infrastructure, but IMO it's non-trivial project.

I think it's well worth the efford (and not that hard). If anybody wants
to try, I'd be willing to help (though you may prefer to take the 
discussion off of this list).

Regards,
                Stefan