Re: [jamboost] Test system

From: David Abrahams (david.abrahams_at_[hidden])
Date: 2002-04-01 12:39:03

Aside from issues of grammar and usage, which we can clear up later:

> Introduction and examples
>
> The test system for Boost.Build is intended to test actions that build
> system does given various build instruction.

I suggest striking the above sentence due to insufficient
informational content.

> It is based on the
> TestCmd Python module used for testing the Scons build system. The
> changes are very few, and I hope that some of them may be adopted by
> TestCmd in future.
>
> The basic steps of testing a build system behaviour are:
>
> Setting the initial working directory state

Suggest: "setting up the filesystem context"

> Running the build system and check:
>
> generated output,
>
> changes made to the working directory,
> new content of the working directory.

Suggest: "changes made to the filesystem"

> Adding, removing or touching files, or changing their content and
then
> repeating the previous step, until satisfied.
>
> For example, suppose that the current working directory contains the
> following files: test1.py, test1/Jamfile, test1/Jamrules and
> test1/foo.cpp.

How can the current working directory contain "test1/foo.cpp"? I think
you should say, "suppose that the current working directory contains a
test file test1.py and a subdirectory test1/ containing Jamfile,
Jamrules, and foo.cpp."

> test1/foo.cpp.
> Suppose also that Jamfile builds an executable foo from
> foo.cpp and that test1.py contains:
>
> import TestBoostBuild
>
> t = TestBoostBuild.TestBoostBuild()

Can we call this thing TestBoostBuild.Tester or something? I hate
redundancy ;-)

> t.copy_tree('test1')

What does the above line do?

> t.run_build_system("-sTOOLS=borland")

Whaat does the above line do? (I know what it does, but a short
comment would help)

>
t.expect_addition(t.mul("bin/foo/borland/debug/runtime-link-dynamic/foo"
, [".exe", ".obj", ".tds"]))

What does the above line do? (you get the picture)

> t.run_build_system("clean")
>
>
t.expect_removal(t.mul("bin/foo/borland/debug/runtime-link-dynamic/foo",
> [".exe", ".obj", ".tds"]))

Use the blank lines for comments which explain what's going on

> This is all needed for a minimal test (and to find a bug in the
> borland toolset!). Running the test1.py gives the following output:
>
> File bin/foo/borland/debug/runtime-link-dynamic/foo.tds not
removed as expected
> FAILED test of D:\MyDocu~1\Work\build\boost-build\boost-build -d0
> at line 144 of TestBoostBuild.py (expect_removal)
> from line 12 of test1.py

Cool! You might want to model some of the architecture on Boost.Test. It
has
ways to specify that particular tests are critical (i.e. should halt
the test on failure) or not.

> Overview of the most important methods of class TestBoostBuild
> follows.
>
> Changing the working directory
>
> The class TestBoostBuild creates a temporary directory in its
> constructor and changes to that directory. It can be modified by
> calling these methods:
>
> copy_tree -- sets the working tree from existing directory

The name copy_tree and the description seem contradictory. The
description makes it sound like the working directory is simply
changed to be the same as the directory specified (no copying).

> TestCmd.write -- sets the content of file in a working directory

Isn't this the same as TestBoostBuild.write? If so, can't we drop the
TestCmd. qualification?

> touch -- changes the modification times of a file

OK

> Examining the working directory and changing it
>
> The method read, inherited from the TestCmd class, can be used to
> read any file in the working directory and check its
> content. TestBoostBuild adds another method for tracking
> changes. Whenever build system is run (via run_build_system), the
> state of working dir before and after running is recorded.

Just the working directory, or the whole directory tree?

> In addition, difference between the two states -- i.e. lists of
> files that were added, removed, modified or touched -- is stored in
> two member variables, tree_difference and unexpected_difference. The
> meaning of latter is to contain differences that are not yet
> accounted for.

What does "accounted for" mean?

> Writer of tests then removes from
> unexpected_difference elements which are OK, for example, if file
> foo was added, a call to expect_addition("foo") will remove that
> file from the list of unexpected additions.

Oh, I think I see, after many re-readings. I think you described this
in the wrong order. I think you should say,

"...and after running is recorded.

The test writer then calls methods such as expect_addition("foo.o")
to denote expected changes in the tree. When all expected changes
have been registered, the "unexpected_difference" attribute will
contain a list of all changes not yet accounted for."

BTW, how is a difference recorded? It can't just be a list of names?
Is it three lists of names?

> Reference documentation
>
> The entire test system is composed of single class TestBoostBuild,
> derived form TestCmd.TestCmd. The methods defined in TestBoostBuild
> are described below.
>
> Method __init__(self, *kw)
> Effects:
>
> Remembers the current working directory in member original_workdir.
>
> Passes tw to the base method __init__, after applying default values
> to elements "program", "match" and "program", setting "interpreter" to
> empty string.

Considering that there is no TestCmd documentation describing what
these parameters do, I don't think this description is adequate.

> Changes current working dir to self.workdir

Similarly, there's no documentation of self.workdir

> Method copy_tree(self, tree_location)
> Effects:
>
> Replaces the current working dir with content of directory at
> tree_location. If tree_location is not absolute pathname, it will be
> treated as relative to self.original_workdir.

So this does, essentially, "rm -rf * ; cp -r tree_location ." ?

If so, you should say, "ERASES the contents of the current working
directory and copies the contents of 'tree_location' there", just to
be clear how dangerous it is.

> Method touch(self, names)
> Effects:
>
> Sets the access and modification times for all files in names to the
> current time. All the elements names should be relative paths.

OK. I think Scons will need the ability to change file contents in
some trivial way, since it can detect identical-ness.

> Method run_build_system(self, extra_args='', stdout=None, stderr='',
status=0, **kw)
> Effects:
>
> Stores the state of the working directory in self.previous_tree.
>
> Calls TestCmd.run, passing it the right program name, and the argument

What is the "right" program name? Again, TestCmd docs are missing.

> Checks the stdout, stderr and exit status, if appropriate arguments
> are not None

How does it respond to these (IOW, what does it mean to "check"?)

> Stores the new state of the working directory in self.tree. Computes
> the difference between previous and current trees and store them in
> variables self.tree_difference and self.unexpected_difference.

Details, please? How are they stored?
Or else, please provide a high-level interface which obviates the need
for me to know the details.

> Methods for declaring expectations
>
> Accordingly to the number of changes kinds that are detected, there
> are four methods that specify that test author expects a specific
> change to occur. They check self.unexpected_difference, and if the
> change is present there, it is removed. Otherwise, test fails.
>
> Each method accepts a list of names. Those names are always
unix-style, even on other systems.
>
> Note:The mul member might be usefull to create lists of names.
>
> Note:The file content can be examined using TestCmd.read function.
>
> The members are:
>
> expect_addition
> expect_removal
> expect_modificatio
> expect_touch
>
> There's also a member expect_nothing_more, which checks that all the
> changes are either expected or ignored, in other words that
> unexpected_difference is empty by now.

At some point we may need to be able to say "Whatever other arbitrary
files may have been added are expected, but expect no other changes".

Oh, lookee, there it is right below!

> Methods for ignoring changes
>
> There are five methods which ignore changes made to the working
> tree. They silently remove elements from self.unexpected_difference,
> and don't generate error if element is not found. They accept shell
> style wildcard.
>
> The following methods correspond to four kinds of changes:
>
> ignore_addition(self, wildcard)
> ignore_removal(self, wildcard)
> ignore_modification(self, wildcard)
> ignore_touch(self, wilcard)
>
> The method ignore(self, wildcard) ingores all the changes made to
> files that match a wildcard.
>
> Method mul(self, *arguments)
>
> Precondition:Each passed arguments must be either string or a list of
> strings.
>
> Effects:Returns a list of strings which are formed by taking one
> string from each passed argument and joining them. The order of
> strings corresponds to lexicagraphical ordering of sequences of
> indices.
>
> Example:
>
> mul("/home/ghost", [".bashrc", ".bash_profile"])
> will return
> ["/home/ghost/.bashrc", "/home/ghost/.bash_profile"]

IMO it would be better to provide class BoostBuildTest.List, such that:

List(r"this is\ a test") * List(r'.foo .py')
== List(r"this.foo this.py is\ a.foo is\ a.py test.foo test.py")

List(r'this is\ a test')[1] == 'is a'