[boost] [testing] Possible changes to testing..

From: Rene Rivera (grafik.list_at_[hidden])
Date: 2005-03-08 21:16:53

All,

I've been reading all the testing related post and not having time to 
respond to them. So I decided to talk about what might be done to 
improve things in one big posting.

For along time now one of my objectives has been to use BuildBot
(http://buildbot.sf.net/) to improve the management of running the
regression tests. For those who don't feel like reading about that
software here's a quick summary:

It is a client-server based configuration where the test server controls
the test clients telling them what to do for test. In this arrangement
the client do not directly control what they are testing, nor how. All
the clients provide is an execution environment for the testing. The
server is configured to control the various clients as to what they
test, including how to get the source, and when they test. The testing
is change driven instead of time driven as we currently have. This means
that each time a change occurs on the source (CVS in our case) the
server decides to tell the clients to run tests. Because of this direct
communication to the clients the server is able to give direct feedback
as to what the testing is doing. This includes a live display of test
progress, down to a dynamic view of the test log.

Some of the issues and how this step might help. And other changes I've 
thought about might help..

*Reproducibility*

BuildBot controls what the checkouts are and what triggered the build. 
You can see this by selecting one of the yellow "Build #" links on the 
BuildBot display. If there's ever a question about what code produced an 
error one can get the specific version of the tree to attempt 
reproduction. The history of what tests have run is kept. Which means 
that we can finally answer the dreaded "When did that start breaking?" 
question.

*Scalability*

One big change I'd like to see is the breakup of running tests to make 
it possible for testers to run tests on subsets of libraries, or even 
individual libraries. For example there might be some testers which have 
a special interest in a particular library, Boost.Python comes to mind. 
It would be ideal to make it possible for them to only run those tests, 
and to go through the extra steps of doing the Python setup. Also for 
some popular platforms, it becomes possible to get much faster response 
rates from testing if, for example, we partition the library testing 
space throughout those platforms and they would test the libraries in 
parallel.

For this to happen some significant organizational changes need to 
happen to the tests. As it currently stands such a division is not 
possible because of the way tests are defined and organized. We have a 
single test point, status/Jamfile, which a) points to the rest of the 
test points, libs/*/test/Jamfile, and b) defines it's own tests. The 
problem is that there is a conflict, for some libraries, between the 
tests defined in status/Jamfile, and the tests defined in 
libs/*/test/Jamfile. For example Boost.Config has a reduced set of test 
in status/Jamfile. This situation is likely a reaction to reduce test 
times to something manageable. And I understand that library authors 
would need to have a place to run their own set of comprehensive tests.

My proposal is to create a set of canonical tests that comprise the 
regression testing suite independent of the library author's tests. This 
set of tests would be structured so that it's possible to run each 
libraries tests independent of others. It would be adding this type of 
structure:

boost-root/tests/<library>/Jamfile
boost-root/tests/<library>/<sub-library>/Jamfile
boost-root/tests/<library>/<some-other-grouping>/Jamfile

So for example a tester could say she only wants to test python/*, or 
numeric/ublas/*, etc.

*Fragility*

Simply put because BuildBot exposes the complete testing procedure it 
becomes much easier to see where problems occur in the testing. Also, 
because there is a central server managing the procedures it's more 
likely that problems can be solved at that one location instead of 
opaquely trying to fix the clients. With the, hopefully, additional 
distributed testers it becomes harder for testing to break down completely.

Another aspect we need to address is fragility of the server. Ideally we 
would have multiple BuildBot servers to add redundancy. To make such a 
multi-server, multi-client setup resource efficient we would need to 
manage the distribution of testing between them.

*Resources*

The only predictable way to address the resource usage, is to distribute 
the testing so we can create more capacity. Breaking up the tests is the 
only way I can see to move there. It was already suggested that only 
slicing the testing to single toolsets would help. But that doesn't 
really address the problems. For example it would still be prohibitive 
for me to run tests on my G3 Mac for CW-8.3 because it's a slow machine 
and it would take days to run just one cycle of tests making the results 
useless. But it would be possible for me to run a minimal set of tests, 
for example Boost.Config and other basic tests.

Restructuring also brings up the possibility of moderating the set of 
tests that make it into the regression suite. Right now we are in a 
position of asking library authors to define what the tests for the 
library. We can't moderate what gets into testing, or what needs go out. 
We need some procedure for reviewing tests, and some form of approval to 
get tests into the regression system. I'm not saying that authors would 
not be able to do additional testing, just that we need to define what 
the standard set of tests are so that we can concentrate our testing 
efforts. It would still be possible to set up additional resources to 
run "experimental" tests.

*Response*

The gain from segmentation and distribution of testing is hopefully 
obvious ;-) But another advantage of using BuildBot is that we are not 
tied to waiting for the XSLT processing to see results. Sure the results 
are not going to be as incredibly well organized as the Meta-Comm 
results but they are immediately available. So if there is a significant 
delay in the processing, because of load or breakage, we can still 
continue working.

*Releases*

Managing the testing for a release was brought up a many times. And it's 
clear that requiring testers to do manual changes is just not working. 
For BuildBot having the control of what is tested on the server means 
that at any point one, or some small number, person(s) can make the 
switch to have testing resources devoted to release testing. One 
possibility that the finner grain testing allows for is to limit the 
testing for a release to the required set of toolsets and platforms.

OK, that's enough rambling... I know I haven't mentioned many other 
items raised, particularly about bjam and Boost.Build. I'll leave that 
for another post. I just need to get back to setting up the BuildBot 
server now ;-)

-- 
-- Grafik - Don't Assume Anything
-- Redshift Software, Inc. - http://redshift-software.com
-- rrivera/acm.org - grafik/redshift-software.com - 102708583/icq