Re: [boost] Trunk tests broken

Subject: Re: [boost] Trunk tests broken
From: Robert Ramey (ramey_at_[hidden])
Date: 2010-06-02 16:56:27

Beman Dawes wrote:
> On Wed, Jun 2, 2010 at 1:41 PM, Belcourt, Kenneth <kbelco_at_[hidden]>

First of all, let me apologize for any inconvenience this may have caused.

> bjam shouldn't stop entirely because a file isn't found. At worst,
> that library's tests should fail.

> Can Boost.Build or bjam be modified to be more tolerant of errors
> during regression testing?

This is the perfect example of why I think trunk testing should be changed.

Here is what I propose.

For each library: - e.g. serialization

a) Start with a machine which two separate release and trunk directory 
trees.
Assume that these are named /boost_release and boost_trunk respectively.

b) invoke the following script:

mv boost_release/serialization boost_x/serialization
mv boost_trunk/serialization boost_release/serialization
mv boost_release/archive boost_x/archive
mv boost_trunk/archive boost_release/archive
mv boost_release/libs/serialization boost_x/libs/serialization
mv boost_trunk//libs/serialization boost_release/libs/serialization

We now have a boost_root tree which is the latest release but
has the trunk for the one library being tested.

c) run the test

cd boost_release/libs/serialization/test
bjam ....
# I run ../../../tools/regression/src/library_test.sh ...which is equivalate 
to the above
# but creates a report in the test directory

c) run a script which reversed the changes made in a) above.

The testing of all of boost would be defined as nothing more than
the same procedure applied to each library.

What would this give is?

a) Problems such as the current one would be be limited only to the 
offending library.

b) bugs in one library wouldn't show up as test failures in another library.

c) Testing would be against the "next release".  This would dimish surprises
that might occur when libraries are merged into the release branch.

d) Testing would be MUCH faster.  Currently, if a header in library A 
changes
in the trunk - then ALL the libraries which depend on A are rebuilt and 
tested.
This costs a huge amount of time for almost no benefit.  Under the new 
system,
only the tests for library A would be re-run if library A changes.  Tests 
for all
other libraries which depend upon A would occur when library A is merged
into the release branch.

e) It takes a baby step in the direction of making boost libraries 
independently
deployable.

f) Certain libraries and tools such as bjam and boost.test are used in the
testing process.   When the developer of one of these tools checks in
something that turns out to be an error - then it creates bogus test results
when can hang around for weeks until they are cleared up.  How can
these developers work in such an environment?

Finally, what would it take to implement this? Only a change in the test 
script.
All current boost tools  - bjam, boost test, etc would remain identical.  I 
don't
know how the test matrix generation works - but I suspect that it would
change little if at all (I'm assuming it reviews the bin.v2 directory which 
would
be the same under this proposal).

Who sees anything wrong with this? Better yet, who can see any
advantage in the current setup?

Robert Ramey