[boost] Re: [Boost.Test] New testing procedure?

From: David Abrahams (dave_at_[hidden])
Date: 2004-05-23 02:15:03

"Robert Ramey" <ramey_at_[hidden]> writes:

> Jeff Garland wrote:
>
>> Well as soon as Robert wants to run the torture test he's going to get it
>> at all sites if he controls it via his Jamfile.  So we need some 
>> boost-wide option to define these variations.  Hopefully my other email
>> clarifies the idea.
>
> My concern was raised base on the following scenario:
>
> a) I get things working to taste on the 3 or 4 compilers I have installed.
> b) I check in to CVS
> c) Test is run - takes a long time 
> d) But shows some problems in the 5 compilers I don't have
> e) Now I start to fix stuff - more or less one compiler at a time
> f) Now much computer resource are used to no good purpose.  I can't stop the
> test from running on the compilers that I know will fail anyway.
> g) Now my interest shifts to another compiler and a whole different set of
> wasted effort is done.
> ....

OK, but none of the features suggested so far are going to fix that.
You just want to say "this library/test doesn't work on these
compilers", and as you start to implement workarounds for each
compiler, knock it off the list so it'll be tested.  We currently
already have a way of expressing that "known to not work on X
compiler" idea in the regression system.  We could make the build
system smart about it.  

Probably simpler, though, you could just put something in one of your
library headers that said:

#if BOOST_WORKAROUND(compiler1) || BOOST_WORKAROUND(compiler2) || ...
# error doesn't work yet
#endif

And the tests would take very little time when they are known to fail.
Hmm, that would cause retesting of known-working configurations.  I
guess a change to the build system would be superior.

> q) Now everything is spiffy
> r) test runs - but only those tests that are dependant on something that has
> changed.  The situation is tolerable - until something changes that changes
> the library - then caos
>
> from q onward the problem might be manageable if BJAM was could manage
> dependencies from  are particular x.hpp -> y.cpp -> test.cpp.  

Of course it can and does.

> But as it is now, if one *.cpp file in the library has to be
> rebuilt, all the test that depend on any portion of the library have
> to be run.  

Of course they should.

> I realize this is an unrealistic hope but we're allowed to dream.

I clearly don't know what you're talking about here.

-- 
Dave Abrahams
Boost Consulting
http://www.boost-consulting.com