[Boost-build] Boost Build testing notes.

From: Jurko Gospodnetiæ (jurko.gospodnetic_at_[hidden])
Date: 2007-12-22 17:54:05

   Hi.

   Here are my testing related notes I promised, taken while playing 
around with Boost Build testing.

   I'm also attaching a patch for some minor stylistic changes and typo 
corrections (based on the nightly build collected today from the Boost 
web site). Among other things I made the readme.txt file reference the 
main HTML documentation since it gave only partial information and 
really confused me when I started (I first opened the readme file and 
only later found the HTML docs).

Bug/defect reports:
-------------------

   * When a Jamfile has a syntax error at its very end such as a missing 
space before the final semicolon (e.g. 'echo "crash and burn";') then 
the correct stack information is not displayed and you only get a way 
too terse 'syntax error at EOF' error messages. Then you have to use 
ECHOs and 'binary search' throughout Boost Build code to find out where 
you messed up.

   * There are some stale tests in the test folder that are not included 
in test_all.py at least some of which fail when run directly.
     * 'abs_workdir.py'.
     * 'core_dependencies.py'.
     * 'core_modifiers.py'.
     * 'direct_request_test.py'.
     * 'm1-01.py'.
     * 'm1-02.py'.
     * 'm1-03.py'.
     * 'ordered_properties.py'.
     * 'project_test1.py'.
     * 'railsys.py'.
     * 'test1.py'.
     * 'test2.py'.
     * 'test_all.py'.
     * 'testing_primitives.py'.
     * 'unused.py'.
     * 'v1_testing.py'.

   * run_tests() function in test_all.py has a comment stating that it 
stops when encounters the first failure which is not true.

Feature requests (low priority):
--------------------------------

   * Allow the diffing system based on the DO_DIFF environment variable 
to work with GNU (cygwin) diff which returns 0/1 in case files are 
same/different respectively or 2 in case of trouble. Currently treats 
return value 1 as a failed diff call and displays an error message on 
every success.

Feature requests (very low priority):
-------------------------------------

   * Testing if the current folder path contains spaces.

   * Specifying that testing should fail after the first failed test.

   * Update source code to reference SVN instead of CVS.

   * Several tests (expansion.py, explicit.py & ordered_properties.py) 
have incorrect code comments left over from the template.py script they 
were based on.

   * Possibly rename the 'test_results.txt' file to 'last_failed_test.txt'.

   * Extract the test and the testing system Python scripts into two 
separate folders or establish a different naming convention allowing the 
user to more easily know which scripts contain tests.
     * Test system Python scripts.
       * 'BoostBuild.py'.
       * 'TestCmd.py'.
       * 'tree.py'.
       * 'svn_tree.py'.
       * 'load_dir.py'.
       * 'template.py'.
     * Additional test system files.
       * test_results.txt.
       * test-config.jam.
       * test-config-example.jam.
       * Documentation files.
     * My vote is to refactor the testing folder hierarchy to two 
levels: testing and tests below it. Top level 'testing' folder would 
contain the testing system files including any locally added files by 
the user while  the child 'tests' folder would hold all the test data 
including test specific subfolders where needed.

Documentation notes:
--------------------
   * test_system.html says that Python 2.1 is known to work. Everything 
worked fine for me with Python 2.5.1.
   * DO_DIFF environment variable needs to be documented.
   * Command-line options that need to be documented.
     * All test scripts.
       * --verbose option.
       * --default-bjam.
     * test_all.py script.
       * --extras.
   * 'test_results.txt' file needs to be documented.
   * 'load_dir.py' utility script needs to be documented.
   * 'template.py' example test script needs to be documented.
   * Either rename the main HTML documentation file to index.html or add 
index.html with that forwards the used to the documentation file. It 
took me a while to find it as it does not have a standard name and is 
hidden among all the other test related files.

   Now I'm going to try and prepare the test case for my bug report. 
Wish me luck. :-)

   Hope this helps.

   Best regards,
     Jurko Gospodnetiæ

Left base folder: X:\Boost_Build\20071222_nightly_build\Original\boost-build
Right base folder: X:\Boost_Build\20071222_nightly_build\Modified\boost-build
--- test\BoostBuild.py	2007-12-21 15:33:44.000000000 +-0100
+++ test\BoostBuild.py	2007-12-22 23:15:57.000000000 +-0100
@@ -458,13 +458,13 @@
     
     def fail_test(self, condition, dump_stdio = 1, *args):
         # If test failed, print the difference        
         if condition and hasattr(self, 'difference'):            
             f = StringIO()
             self.difference.pprint(f)
-            annotation("changes causes by the last build command", f.getvalue())
+            annotation("changes caused by the last build command", f.getvalue())
             
         if condition and dump_stdio:
             self.dump_stdio()
 
         if condition and '--preserve' in sys.argv:
             print 
@@ -659,13 +659,12 @@
             print "Got:\n"
             print actual
             self.fail_test(1)
 
     def maybe_do_diff(self, actual, expected):
         if os.environ.has_key("DO_DIFF") and os.environ["DO_DIFF"] != '':
-            
             e = tempfile.mktemp("expected")
             a = tempfile.mktemp("actual")
             open(e, "w").write(expected)
             open(a, "w").write(actual)
             print "DIFFERENCE"
             if os.system("diff -u " + e + " " + a):
--- test\empty.jam	2007-12-21 15:33:44.000000000 +-0100
+++ test\empty.jam	2007-12-22 16:30:07.000000000 +-0100
@@ -1,5 +1,5 @@
-# This file is empty; it just suppresses warnings
-
 # Copyright 2001 Dave Abrahams
 # Distributed under the Boost Software License, Version 1.0.
 # (See accompanying file LICENSE_1_0.txt or http://www.boost.org/LICENSE_1_0.txt)
+
+# This file is empty; it just suppresses warnings
--- test\readme.txt	2007-12-21 15:33:44.000000000 +-0100
+++ test\readme.txt	2007-12-22 15:23:03.000000000 +-0100
@@ -1,7 +1,6 @@
-Comprehensive tests for Boost.Build v2; requires Python. To test, execute:
-
-    python test_all.py
-
 # Copyright 2002 Dave Abrahams
 # Distributed under the Boost Software License, Version 1.0.
 # (See accompanying file LICENSE_1_0.txt or http://www.boost.org/LICENSE_1_0.txt)
+
+  See test_system.html for detailed information on using the Boost Build test
+system.
--- test\test_system.html	2007-12-21 15:33:44.000000000 +-0100
+++ test\test_system.html	2007-12-22 23:01:19.000000000 +-0100
@@ -88,13 +88,13 @@
     <h2><a name="sec-intro">Introduction for users</a></h2>
 
     <p>The testing system for Boost.Build is a small set of Python modules
     and scripts for automatically testing user-obversable behaviour. It uses
     components from testing systems of <a href=
     "http://www.scons.org">Scons</a> and <a href=
-    "http://subversion.tigris.org">Subverion</a>, together with some
+    "http://subversion.tigris.org">Subversion</a>, together with some
     additional functionality.</p>
 
     <p>To run the tests you'd need:</p>
 
     <ol>
       <li>Get the source tree of Boost.Build (located at <tt>tools/build</tt>
--- util\option.jam	2007-12-21 15:33:44.000000000 +-0100
+++ util\option.jam	2007-12-22 18:20:37.000000000 +-0100
@@ -22,13 +22,13 @@
 
 rule process ( )
 { 
     # Check command-line args as soon as possible.  For each option try
     # to load module named after option. Is that succeeds, invoke 'process'
     # rule in the module. The rule may return "true" to indicate that the
-    # regular built process should not be attempted.
+    # regular build process should not be attempted.
     #
     # Options take the general form of: --<name>[=<value>] [<value>]
     #
     
     local ARGV = [ modules.peek : ARGV ] ;
     local BOOST_BUILD_PATH = [ modules.peek : BOOST_BUILD_PATH ] ;