Re: [jamboost] Test system

From: Vladimir Prus (ghost_at_[hidden])
Date: 2002-04-03 06:13:29

> Aside from issues of grammar and usage, which we can clear up later:

BTW, the document is now in CVS under tools/build/test/test_system.html
I'll commit the code later, if we don't have any more issues with semantics 
and docs.

> > Introduction and examples
> >
> > The test system for Boost.Build is intended to test actions that build
> > system does given various build instruction.
>
> I suggest striking the above sentence due to insufficient
> informational content.

Striken. Now the docs have no introduction phrase at all.

> > It is based on the
> > TestCmd Python module used for testing the Scons build system. The
> > changes are very few, and I hope that some of them may be adopted by
> > TestCmd in future.
> >
> > The basic steps of testing a build system behaviour are:
> >
> > Setting the initial working directory state
>
> Suggest: "setting up the filesystem context"

> > Running the build system and check:
> >
> > generated output,
> >
> > changes made to the working directory,
> > new content of the working directory.
>
> Suggest: "changes made to the filesystem"
>

> > Adding, removing or touching files, or changing their content and
>
> then
>
> > repeating the previous step, until satisfied.
> >
> > For example, suppose that the current working directory contains the
> > following files: test1.py, test1/Jamfile, test1/Jamrules and
> > test1/foo.cpp.
>
> How can the current working directory contain "test1/foo.cpp"? I think
> you should say, "suppose that the current working directory contains a
> test file test1.py and a subdirectory test1/ containing Jamfile,
> Jamrules, and foo.cpp."

Reworded.

> > test1/foo.cpp.
> > Suppose also that Jamfile builds an executable foo from
> > foo.cpp and that test1.py contains:
> >
> > import TestBoostBuild
> >
> > t = TestBoostBuild.TestBoostBuild()
>
> Can we call this thing TestBoostBuild.Tester or something? I hate
> redundancy ;-)

I've changed this to BoostBuild.Tester. Hope nobody will mistake "BoostBuild" 
python module for the actual build system.

> > t.copy_tree('test1')
>
> What does the above line do?
>
> > t.run_build_system("-sTOOLS=borland")
>
> Whaat does the above line do? (I know what it does, but a short
> comment would help)
>
>
> t.expect_addition(t.mul("bin/foo/borland/debug/runtime-link-dynamic/foo"
> , [".exe", ".obj", ".tds"]))
>
> What does the above line do? (you get the picture)
>
> > t.run_build_system("clean")
>
> t.expect_removal(t.mul("bin/foo/borland/debug/runtime-link-dynamic/foo",
>
> > [".exe", ".obj", ".tds"]))
>
> Use the blank lines for comments which explain what's going on

I've done so.

> > This is all needed for a minimal test (and to find a bug in the
> > borland toolset!). Running the test1.py gives the following output:
> >
> > File bin/foo/borland/debug/runtime-link-dynamic/foo.tds not
>
> removed as expected
>
> > FAILED test of D:\MyDocu~1\Work\build\boost-build\boost-build -d0
> > at line 144 of TestBoostBuild.py (expect_removal)
> > from line 12 of test1.py
>
> Cool! You might want to model some of the architecture on Boost.Test. It
> has
> ways to specify that particular tests are critical (i.e. should halt
> the test on failure) or not.

I'm not sure. I still belive that using QMTest will be reasonable. If we use 
it, it will be possible to specify "prerequisite tests" for any given tests. 

> > Overview of the most important methods of class TestBoostBuild
> > follows.
> >
> > Changing the working directory
> >
> > The class TestBoostBuild creates a temporary directory in its
> > constructor and changes to that directory. It can be modified by
> > calling these methods:
> >
> > copy_tree -- sets the working tree from existing directory
>
> The name copy_tree and the description seem contradictory. The
> description makes it sound like the working directory is simply
> changed to be the same as the directory specified (no copying).

The new wording and method name is:
set_tree -- sets the content of the working directory to be equal to the 
content of the specified directory.

> > TestCmd.write -- sets the content of file in a working directory
>
> Isn't this the same as TestBoostBuild.write? If so, can't we drop the
> TestCmd. qualification?

Wanted to emphasize that the method comes from TestCmd, but looks like it 
will only confuse users, especially since we don't have any docs to TestCmd, 
except for comments in code.

> > Examining the working directory and changing it
> >
> > The method read, inherited from the TestCmd class, can be used to
> > read any file in the working directory and check its
> > content. TestBoostBuild adds another method for tracking
> > changes. Whenever build system is run (via run_build_system), the
> > state of working dir before and after running is recorded.
>
> Just the working directory, or the whole directory tree?

Does "the state of working directory (and all its subdirectories)" makes more 
sense?

> > In addition, difference between the two states -- i.e. lists of
> > files that were added, removed, modified or touched -- is stored in
> > two member variables, tree_difference and unexpected_difference. The
> > meaning of latter is to contain differences that are not yet
> > accounted for.
>
> What does "accounted for" mean?

Hopefully clarified. 

> > Writer of tests then removes from
> > unexpected_difference elements which are OK, for example, if file
> > foo was added, a call to expect_addition("foo") will remove that
> > file from the list of unexpected additions.
>
> Oh, I think I see, after many re-readings. I think you described this
> in the wrong order. I think you should say,
>
> "...and after running is recorded.
>
> The test writer then calls methods such as expect_addition("foo.o")
> to denote expected changes in the tree. When all expected changes
> have been registered, the "unexpected_difference" attribute will
> contain a list of all changes not yet accounted for."

Does the new version make it more clear?

> BTW, how is a difference recorded? It can't just be a list of names?
> Is it three lists of names?
>
> > Reference documentation
> >
> > The entire test system is composed of single class TestBoostBuild,
> > derived form TestCmd.TestCmd. The methods defined in TestBoostBuild
> > are described below.
> >
> > Method __init__(self, *kw)
> > Effects:
> >
> > Remembers the current working directory in member original_workdir.
> >
> > Passes tw to the base method __init__, after applying default values
> > to elements "program", "match" and "program", setting "interpreter" to
> > empty string.
>
> Considering that there is no TestCmd documentation describing what
> these parameters do, I don't think this description is adequate.

Neither do I. I have removed any mentioning about various values, becase 
using them is hardly ever needed.

> > Changes current working dir to self.workdir
>
> Similarly, there's no documentation of self.workdir

Fixed.

> > Method copy_tree(self, tree_location)
> > Effects:
> >
> > Replaces the current working dir with content of directory at
> > tree_location. If tree_location is not absolute pathname, it will be
> > treated as relative to self.original_workdir.
>
> So this does, essentially, "rm -rf * ; cp -r tree_location ." ?
>
> If so, you should say, "ERASES the contents of the current working
> directory and copies the contents of 'tree_location' there", just to
> be clear how dangerous it is.

Isn't "replaces" quite explicit already?

> > Method run_build_system(self, extra_args='', stdout=None, stderr='',
>
> status=0, **kw)
>
> > Effects:
> >
> > Stores the state of the working directory in self.previous_tree.
> >
> > Calls TestCmd.run, passing it the right program name, and the argument
>
> What is the "right" program name? Again, TestCmd docs are missing.
>
> > Checks the stdout, stderr and exit status, if appropriate arguments
> > are not None
>
> How does it respond to these (IOW, what does it mean to "check"?)

I've clarified this.

>
> > Stores the new state of the working directory in self.tree. Computes
> > the difference between previous and current trees and store them in
> > variables self.tree_difference and self.unexpected_difference.
>
> Details, please? How are they stored?

Wrote.

> Or else, please provide a high-level interface which obviates the need
> for me to know the details.

The details are too simple to create an additional interface layer.

> > Methods for declaring expectations
> >
> > Accordingly to the number of changes kinds that are detected, there
> > are four methods that specify that test author expects a specific
> > change to occur. They check self.unexpected_difference, and if the
> > change is present there, it is removed. Otherwise, test fails.
> >
> > Each method accepts a list of names. Those names are always
>
> unix-style, even on other systems.
>
> > Note:The mul member might be usefull to create lists of names.
> >
> > Note:The file content can be examined using TestCmd.read function.
> >
> > The members are:
> >
> > expect_addition
> > expect_removal
> > expect_modificatio
> > expect_touch
> >
> > There's also a member expect_nothing_more, which checks that all the
> > changes are either expected or ignored, in other words that
> > unexpected_difference is empty by now.
>
> At some point we may need to be able to say "Whatever other arbitrary
> files may have been added are expected, but expect no other changes".
>
> Oh, lookee, there it is right below!
>
> > Methods for ignoring changes
> >
> > There are five methods which ignore changes made to the working
> > tree. They silently remove elements from self.unexpected_difference,
> > and don't generate error if element is not found. They accept shell
> > style wildcard.
> >
> > The following methods correspond to four kinds of changes:
> >
> > ignore_addition(self, wildcard)
> > ignore_removal(self, wildcard)
> > ignore_modification(self, wildcard)
> > ignore_touch(self, wilcard)
> >
> > The method ignore(self, wildcard) ingores all the changes made to
> > files that match a wildcard.
> >
> > Method mul(self, *arguments)
> >
> > Precondition:Each passed arguments must be either string or a list of
> > strings.
> >
> > Effects:Returns a list of strings which are formed by taking one
> > string from each passed argument and joining them. The order of
> > strings corresponds to lexicagraphical ordering of sequences of
> > indices.
> >
> > Example:
> >
> > mul("/home/ghost", [".bashrc", ".bash_profile"])
> > will return
> > ["/home/ghost/.bashrc", "/home/ghost/.bash_profile"]
>
> IMO it would be better to provide class BoostBuildTest.List, such that:
>
> List(r"this is\ a test") * List(r'.foo .py')
> == List(r"this.foo this.py is\ a.foo is\ a.py test.foo test.py")
>
> List(r'this is\ a test')[1] == 'is a'

Why would you like it? When specifying name to {except, ignore}* methods, it 
make no difference. Or do you have wider applications for such class?

I also would like to thank you for all the suggestions you've made above. 
They really help.

- Volodya