Re: [jamboost] On testing

From: Vladimir Prus (ghost_at_[hidden])
Date: 2002-03-26 10:46:21

Steven Knight wrote:

> > > What to do with the second component? We most likely will need
> > > something more usable. If I was to choose, I'll simple use QMTest
> > > (http://www.qmtest.com) for this. It is quite functional, quite easy
> > > to use, and is written in Python -- i.e. test code for the build
> > > system will be easy to integrate with it. On the other hand, it's
> > > more than a couple of modules -- i.e. if somebody wants to test
> > > build system, he'll need to get QMTest. I actually don't think this
> > > is big problem. So, what other's opinions on this?
> >
> > The Scons project has a mature python-based system for testing that
> > has worked well for them. Should we reinvent the wheel, or would it be
> > better to just hijack the work they've done?
>
> You'd be more than welcome to do so, if I'd you'd like. I'd be happy to
> provide code, doc, advice, etc. in any way that's useful.
>
> I had a hard time getting excited about QMTest, personally. IIRC, it
> struck me as a very generic test execution and results-gathering and
> reporting framework. That's fine so far as it goes, but it solves the
> easy part of the problem. It still leaves you to create all of the
> actual infrastructure for your testing environment, the stuff that
> determines whether it's difficult or easy for Joe Developer to actually
> *write* a test.

This is true, QMTest solves the generic problem but I does solve it, and I 
like the solution. For example, I believe that making a QMTest setup for 
Scons would be extra simple. (Okay... I will actually try it!)

> > I'm inclined to go with whatever seems most natural to you, *as long
> > as* there's an easy way for the rest of us to learn how to use it. It
> > would be worth hearing what Steven Knight has to say about the Scons
> > stuff, though.
>
> Here's an overview:

I should say that have looked at Scons only today, and I think that I like 
the test code. I'm not sure if I like it better that what I have in mind (and 
which is close to what I have). 

> In the SCons testing infrastructure, all tests are self-contained Python
> scripts that execute tests in one or more temporary directories. Any
> necessary files are created from in-line Python strings.

I like the idea of using actual directory layout for specifying tree for a 
simple reason. I can just create the tree on disk, play with it, and the 
convert it into test with no effort. Or later, I can easily play with the 
tree that a test uses, in case the test fails.

> There is one underlying generic TestCmd.py module that provides
> primitives for creating and removing temporary directories, writing
> files, touching files, comparing actual and expected output, reporting
> PASSED, FAILED, or NO RESULT, etc.

Actually, when I first seen your code I thought: "Oh.. here a file is simply 
read and its content is tested with expected, while my code needs to build 
some tree first, just to do the same!". My second thought is: does you code 
allow to detect when a file was added/removed and, more importantly, when it 
was touched/modified. I'm not sure this is straightforward -- attempt to open 
an nonexisted file will just raise IOError, and you can't predict an expected 
content of a binary file. One example is when static library was not updated 
on borland: the test needed to touch one source and then check if the library 
is touched.

> The feedback from the developers has been good. They've all found this
> framework makes writing SCons tests easy enough that they all include
> new or modified tests in their patches; I haven't had to crack down or
> bug anyone to provide tests. (Of course, it helps that they *do* know
> that tests are expected if they want their patch integrated... :-)

Sure :-) Actually, it should be noted that the code which actually senses 
which files are added/changed etc. is "stolen" from SCM tool Subversion 
(http://subversion.tigris.org) where it's used quite actively to write tests, 
and seems like developers don't object either.

> If it looks like you wanted to adopt any of this for Boost, the right
> way would be to create a TestBoost.py subclass of TestCmd.py that you'd
> tailor to the specific needs of Boost (the functionality Vladimir
> alluded to above). I've been meaning to put TestCmd.py out on the
> Vaults of Parnassus anyway, but haven't had time with trying to bring
> the SCons feature set up to modern standards. This would certainly
> provide the right incentive...
>
> Even if you go with something else, I hope this has provided some useful
> ideas. If I can answer any follow-on questions, let me know.

Can only hope we'll agree on this soon. 

- Volodya