[boost] Fast-test-cycle Proposal [was 1.31.0 Progress (or lack thereof)]

From: Beman Dawes (bdawes_at_[hidden])
Date: 2004-01-26 11:02:37

[Discussion and rationale follow the proposal itself]

Fast-test-cycle Proposal
------------------------

* No changes to current regression tests or their schedule.
* Add a new quick-test jamfile:
     -- One permanent test, designed to spot trouble in key libraries
        many other Boost libs depend on, such as iterator adaptors and
        type traits.
     -- Must compile and run very quickly (say less than 10 seconds
        per compiler) so that it can be cycled quickly.)
     -- Boost developers temporarily add other tests when they need
        quick verification that a change works for compilers not
        available to them.
* Keep the reporting very simple (to speed cycle and upload times)
* Run the quick-test very often - once an hour or more during heavy 
activity. (Presumably on Win32, because GCC on Win32 is also indicative of 
GCC results on other platforms.)
* In the future, if quick-test is found to be useful, dedicate a machine to 
run quick-test continuously.

Would this be useful and worthwhile?

--Beman

Discussion and rationale
------------------------

At 06:20 PM 1/25/2004, Walter Landry wrote:

 >Rene Rivera <grafik.list_at_[hidden]> wrote:
 >> Beman Dawes wrote:
 >>
 >> > I'm basically in a holding pattern on the release until (1)
 >> > SourceForge clears the CVS performance issues, and (2) the above
 >> > regressions get fixed.
 >>
 >> Aren't we all... I was hoping to make a release of Boost.Jam (3.1.9)
 >> today to match the eventual Boost release. But given that updating
 >> CVS takes 4 to 5 hours I don't think that will happen :-(
 >
 >Perhaps this isn't the best time to raise this, but wouldn't it be
 >better if Boost eventually moved to a more distributed version control
 >system?  Then there wouldn't be a single point of failure that stopped
 >all work.  I'm not entirely sure that there is a system which has
 >everything that Boost needs (e.g. free, distributed, runs natively on
 >Windows, and can work with the current Sourceforge infrastructure),
 >but it might be useful to consider alternatives.

It is certainly clear that we will have to make some changes to smooth the 
release process if only to retain the release manager's sanity. Not to 
mention allowing Boost to grow. There was some recent discussion of 
breaking Boost into sections or sub-sets, but I'd like to ignore that for 
now and just concentrate on version control and testing.

At first I also thought that a change to how we do version control would be 
a big step forward. I was thinking in terms of two-step commits so that 
preliminary commits don't impact the main trunk in cases where a fix is 
broken. I'm under the impression that several large open source projects 
have moved to that model, but I don't know any details.

But then over the last several weeks I watched how actual problems arose, 
were identified, and then finally fixed. It seemed to me that several 
particular situations need to be addressed, and it might be better to 
address them directly (see below) rather than indirectly by changing how we 
do version control:

1) A developer having access to only compiler x commit changes which work 
with x but break with compiler y and z. By the time our 24-hr test cycle 
completes, and the developer is busy or forgets to check, etc, etc, it 
takes a week or more to clean up the mess, and may well waste list and 
release manager bandwidth.

2) Many Boost libraries are dependent on a few key Boost libraries like 
type traits and iterator adaptors. If a change to a key library is broken 
for a compiler, many other libraries become untestable until the key 
library is fixed. Regression testing takes much longer (because of massive 
recompilation) until the key library is stablized.

3) (1) and (2) sometimes happen together.

It seems these are testing issues rather than version control issues.

The best way to improved testing is for developers to test on more 
compilers BEFORE they commit code to CVS. Second best might be to markedly 
speed the commit-test-report cycle for (1) and (2).

See the proposal above.

--Beman