[boost] Re: [Boost.Test] New testing procedure?

From: David Abrahams (dave_at_[hidden])
Date: 2004-05-23 01:38:43

"Jeff Garland" <jeff_at_[hidden]> writes:

>> > Well as soon as Robert wants to run the torture test he's going to get 
>> > it at all sites if he controls it via his Jamfile.  So we need some
>> > boost-wide option to define these variations.  Hopefully my other 
>> > email clarifies the idea.
>> 
>> It does, although I don't see how we can manage/afford all the combinations.
>
> I agree.  The combinatorics aren't good, and I even forgot to include the
> debug/release dimension.  

Tests aren't generally designed to be run in release mode.  In
release mode, assertions are turned off.

> But this is where additional testers could help.  One group could
> run with release builds while another runs debug builds.
>
> So the current state of affairs is that there isn't a standard set of flags to
> control the debug/release and linking options.  

There certainly is, for debug/release; you stick it in the BUILD
variable.

> So if the library author wants static linking he basically has to
> define static linking in the Jamfile.  What I'm thinking is that
> control of the linking and debug/release options shouldn't have to
> be specified this way.  

Why do you think that?  I think your whole view of the static/dynamic
issue is naive.  Static and dynamic objects can't always be built the
same way, so that may have to be part of the Jamfile.  And
static/dynamic linking isn't an all-or-nothing proposition.  Every
library test involves at least two libraries (the one being tested and
the runtime), sometimes more.  There's no inherent reason some
couldn't be statically linked and others dynamically linked.

Furthermore, I don't see a big advantage in having a separate
command-line option to choose which of those linking modes is used.
If the library needs to be tested in several variants, then so be it.
If it doesn't, but you'd like to see more variants sometimes, you can
put some of the variants into the --torture option.

> The same set of tests should be runnable by the regression tester
> and hence associated with the column in the results table.  So
> something like:
>
>        VC7.1     VC7.1    VC7.1  etc
>        release   release  debug
>        dll       static   dll
>
> test1  Pass      fail    Pass
> test2  Pass      Pass    Pass
> ...
>  
> Then basic/complete option controls the number of tests run.

That's outta hand, IMO.  If it's worth having options, let's keep
them simple.

-- 
Dave Abrahams
Boost Consulting
http://www.boost-consulting.com