[Boost-testing] Can we improve throughput on our testing?

From: Doug Gregor (dgregor_at_[hidden])
Date: 2005-06-14 15:58:58

Regression testing is most effective when we can immediately see the 
results of changes to the repository. We know that "immediately" isn't 
really possible, because Boost takes a long time to build & test and we 
have limited resources. However, I'd like to shoot for one-day 
turnaround on each of our primary platforms if possible.

To do this, I think we need to try to balance the load a bit. The 
MetaComm and Martin Wille tests cover a huge number of compiler 
variants, but they also have 2-day turnaround times which makes it 
harder to find and fix errors. Let's communicate amongst ourselves to 
divide up the set of tests that each tester runs and try to get in the 
daily results by, say, 10am EST.

For instance, our Linux box (OSL2) is now running gcc-3.3.6-linux and 
gcc-3.4.4-linux nightly, submitting around :00am EST each day, so we 
could speed up the Martin Wille tests a bit by dropping those toolsets. 
We could also add one other toolset to OSL2, but only if it doesn't 
push back the submission time too far. If it does, we'll bring in 
another Linux box (OSL3) to do the testing.

If need be, we could drop testing of minor variants (gcc-3.3.5-linux 
vs. gcc-3.3.6-linux) to get better throughput.

My point is simple: More testing is good, but predictable, up-to-date 
results are better.

What say you?

        Doug