Re: [jamboost] On testing

From: Steven Knight (knight_at_[hidden])
Date: 2002-03-27 09:39:27

> > > My second thought is: does you
> > > code allow to detect when a file was added/removed
> >
> > That would typically be handled by something like:
> >
> > test.run()
> > test.fail_test(os.path.exists(test.workpath('should_not_exist')))
> > test.fail_test(not os.path.exists(test.workpath('should_exist')))
> 
> But... this will detect if a file is present, not if it was *added*.

Good point, but easily solvable:

test.no_result(os.path.exists(test.workpath('should_be_added')
test.run()
test.fail_test(not os.path.exists(test.workpath('should_be_added'))

> > > and, more importantly, when
> > > it was touched/modified. I'm not sure this is straightforward -- attempt
> > > to open an nonexisted file will just raise IOError,
> [ snip] 
> > So far we've used os.path.getmtime() successfully:
> >
> > oldtime = os.path.getmtime('foo')
> > test.run()
> > test.fail_test(oldtime != os.path.getmtime('foo'))
> 
> Not that simple: you need a call to test.workpath in this case as well :-)

You're right for the general case of using TestCmd.py directly. But
the SCons tests can cheat on this one because the TestSCons.py subclass
changes to the temporary directory.

> True, this works but I'd rather have a separate method for this.

I agree with this. In general, there should be more in the way of
wrapper methods in TestCmd.py instead of direct use of os.path.*(). It
makes it that much easier to maintain portability going forward.

> > Note that you can run() an arbitrary program by using the appropriate
> > keyword argument.
> >
> > TestCmd.py also supports use of regular expression matches on output,
> > instead of exact matches, which comes in handy when we're checking
> > output where the line numbers vary, for example.
> 
> These are good things. I'm currently leaning towards the following solution. 
> I'll take scons code and then will add a facility for remembering and 
> comparing the complete state of tree before and after build system 
> invocation. (e.g there will be no need for getmtime calls). There's a 
> question if it worth the trouble -- for me, the fact that I'll reuse existing 
> code settles the question. 
> 
> In detail, I will:
> 1. Add a method to copy a an existing tree to the temporary area
> 2. Add a methon "run_build_system" which will 
> a) call 'run' method
> b) traverse that temporary area and create in-memory tree
> representation
> c) Compare the tree representation with the previous one, to
> find additions/touches etc.
> 3. Add a number of methods which test tree differences against
> expected, for example:
> expect_addition
> expect_touch
> expect_nothing_more

I like the sound of all of this a lot. I think the addition of methods
to make it easier to work with an external directory tree in the way you
describe would be extremely valuable. I'd be very glad to work with
you on this and fold any of your method that apply to the underlying
functionality into TestCmd.py, in an anticipation of releasing it for
public consumption on the Vaults of Parnassus.

I've always intended for TestCmd.py to be a very generic interface that
would support multiple testing methodologies. It's just that it's not
hard to lose track of that goal when you only have your own testing
methodology to support... :-)

> I have to observe that this would make use of direct filesystem operations a 
> bad style, bypassing the primary method, but I think this is OK. 

Per the above, I actually think bypassing the the primary methods
would actually be a good thing in the long run.

Let me know if there's anything I can do to help in the short term.

--SK