Re: [boost] Fast-test-cycle Proposal [was 1.31.0 Progress (or lack thereof)]

From: Brian McNamara (lorgon_at_[hidden])
Date: 2004-01-27 04:36:07

As someone eminently _under_qualified to comment on this proposal, here
are my two bits.  :)

On Mon, Jan 26, 2004 at 11:02:37AM -0500, Beman Dawes wrote:
> [Discussion and rationale follow the proposal itself]
> 
> Fast-test-cycle Proposal
> ------------------------
> 
> * No changes to current regression tests or their schedule.
> * Add a new quick-test jamfile:
>     -- One permanent test, designed to spot trouble in key libraries
>        many other Boost libs depend on, such as iterator adaptors and
>        type traits.

This sounds useful, both to address your point #2:

> 2) Many Boost libraries are dependent on a few key Boost libraries like 
> type traits and iterator adaptors. If a change to a key library is broken 
> for a compiler, many other libraries become untestable until the key 
> library is fixed. Regression testing takes much longer (because of massive 
> recompilation) until the key library is stablized.

and also maybe to sow the seeds that may address

> ... There was some recent discussion of 
> breaking Boost into sections or sub-sets, but I'd like to ignore that for 
> now and just concentrate on version control and testing.

That is, the libraries covered under the the proposed quick-test would
probably constitute "boost-core".  (I have never taken a hard look at the
dependency tree to know the truth, but I suspect that indeed a small
"core" subset of libraries account for a large portion of dependencies.)

>     -- Must compile and run very quickly (say less than 10 seconds
>        per compiler) so that it can be cycled quickly.)
>     -- Boost developers temporarily add other tests when they need
>        quick verification that a change works for compilers not
>        available to them.

This last bullet also sounds very useful.  

I know almost nothing about the capabilities of jamfiles and Boost test
automation, but it sounds like it would be swell if you could
commit/submit such a "temporary" test and have these temporary tests
automatically remove themselves after the next successful test cycle
completes (so that temporary tests don't get forgotten and
accumulate).

...
> At first I also thought that a change to how we do version control would be 
> a big step forward. I was thinking in terms of two-step commits so that 
> preliminary commits don't impact the main trunk in cases where a fix is 
> broken. I'm under the impression that several large open source projects 
> have moved to that model, but I don't know any details.
> 
> But then over the last several weeks I watched how actual problems arose, 
> were identified, and then finally fixed. It seemed to me that several 
> particular situations need to be addressed, and it might be better to 
> address them directly (see below) rather than indirectly by changing how we 
> do version control:
...
> It seems these are testing issues rather than version control issues.

Kudos for this analysis.

> The best way to improved testing is for developers to test on more 
> compilers BEFORE they commit code to CVS. Second best might be to markedly 
> speed the commit-test-report cycle for (1) and (2).
> 
> See the proposal above.
> 
> --Beman

-- 
-Brian McNamara (lorgon_at_[hidden])