On testing

From: Vladimir Prus (ghost_at_[hidden])
Date: 2002-03-26 05:43:13

Since I feel a bit unsure about committing untested code, I'd like to discuss 
how the testing will be performed. 

First there's code that is capable of running build system and finding which 
files are added/removed &c. It also allows to change file content and 
rebuild, and similar operations. It's written in Python and it was agreed 
that translating it in Jam is hard and unneeded. Does everybody agree with it.

Second, the code to run individual tests. Now I have a naive code based on 
Python unit tests and it even does not allow to run any individual test, 
IIRC. I really don't want to do anything about it -- it was a temporary 
solution. 

The questions is:

What to do with the second component? We most likely will need something more 
usable. If I was to choose, I'll simple use QMTest (http://www.qmtest.com) 
for this. It is quite functional, quite easy to use, and is written in Python 
-- i.e. test code for the build system will be easy to integrate with it. On 
the other hand, it's more than a couple of modules -- i.e. if somebody wants 
to test build system, he'll need to get QMTest. I actually don't think this 
is big problem. So, what other's opinions on this? 

For those who never seen the my test code I'd outline my vision of how the 
test suite might look. It will be a directory hierarchy (corresponding to 
tests hierarchy) and at some level we might have this:

<some directory>
test1.qmt
test2.qmt
test1-tree
test2-tree

*.qmt are files that descibe tests and test{1,2}-tree are just directories. 
They will contain initial state of source tree for tests. Inside *.qmt files, 
python code will be used to drive tests, something like
touch_file("src/source.cpp")
run_build_system()
expect_touch("build/main/release/runtime-link-dynamic/source.o")

- Volodya