Re: [jamboost] Separation of testing front- and back-ends.

From: David Abrahams (david.abrahams_at_[hidden])
Date: 2002-01-14 10:28:03

Brad,

This is great. My first impression is that this is a /lot/ more like what
the build system rewrite is supposed to be than what it currently is. The
fact that you came up with a similar declarative design approach gives me a
little confidence as well ;^)

We now have 2 choices:
1. Try to integrate this with the current Boost.Build
2. Push forward with the rewrite

I tend to favor #2, but am open to arguments. My reasons are:

a. the structure of module-based Boost.Jam code is so very different from
the big soup we currently have.
b. the current approach of repeatedly reloading toolset definitions for each
target has a high cost. Building the same test via different toolsets AND
different testing modules will only explode the problem
c. I'd like to invest as little as neccessary in the existing codebase, and
move on to the rewrite ASAP.

> Developers declare their tests like this:
>
> test.compile test1 : source1 ;
> test.compile-fail test2 : source2 ;
> test.link test3 : source3 ;
> test.link-fail test4 : source4 ;
> test.run test5 : source5 : : -foo ;
> test.run-fail test6 : source6 : : -bar : test.in ;

That's basically our existing syntax; super. You should look at the
python.jam file too, though, since there are lots of tests in there as well.
It's not well-integrated with the other testing code; I'm sure there's some
duplicated functionality there. Your testing system should make it possible
to add new kinds of tests (e.g. python-run), or possibly provide a framework
through which anything can be run.

> After all these tests are declared (after the user jamfile has been
> included, for example), these lines appear:
>
> # Let the user list the tests with "jam list-tests"
> test.list list-tests ;

I /love/ this idea! I never would have thought of it.

> # Invoke the "demo" testing module for "jam demo-test".
> demo.invoke demo-test ;
>
> The demo.invoke rule just forwards its target to a "test.invoke" rule
> which then calls all the test rules with "demo." as a prefix instead of
> "test." as a prefix, and with all the same arguments. The test.invoke
> method also creates a target for each test automatically. Each of demo's
> rules can refer to $(test-target) to set up the target.
>
> This way, the user can update a specific test by saying:
> jam demo-test.test1
> jam demo-test.test2
> ...
> or update all tests by saying:
> jam demo-test

Nice interface...

I hope we can also have "jam test1", which runs the test by the user's
preferred testing means.

> To see what a particular test entails, the user can also list a specific
> test:
> jam list-tests.test1
> jam list-tests.test2

What's that going to tell you? The command-line that will get executed
perhaps?

> ...
> or list all tests by saying:
> jam list-tests
>
> This approach has the following advantages:
>
> 1.) Developers specify their test only once.
> 2.) Testing back-ends can be added without changing the "test" module.
> 3.) Users can choose exactly what back end is to be used, and specify
> either a specific test or all tests.
> 4.) Back-end authors can write their rules almost as if they were being
> called directly, but they don't have to worry about the test target
> name...they just use $(test-target).

Some other features supported by the current system:

"jam <testname>.run" will run the test even if there's an up-to-date record
of its success. Now that I think of it, I wonder if it woulnd't be better if
"jam <testname>" had that behavior, while "jam test" or "jam test-update"
would only run outdated tests.

If you look at the python.jam file, you'll see that there's a PYTHON_LAUNCH
variable which can be used to say how python is invoked. I commonly use this
to run a debugger in the same context in which the test needs to be run.

While your approach is basically sound, it will need some adjustment to be
compatible with the planned rewrite. Some things I noticed:

1. We don't write "module" explicitly, except in low-level code. See the
contents of tools/build/new, and especially modules.jam

2. Part of the plan is to delay generation of targets (meaning the use of
DEPENDS and action rules) until after the Jamfile has been completely read
in. There are lots of good reasons for this, which you can read about in the
message history. So, your initial level of indirection/delay will have to be
extended.

Again, thanks for your contribution; it's great to have you on board!

-Dave