Re: [boost] Re: [Boost.Test] New testing procedure?

From: Jeff Garland (jeff_at_[hidden])
Date: 2004-05-23 11:49:26

On Sun, 23 May 2004 02:38:43 -0400, David Abrahams wrote
> "Jeff Garland" <jeff_at_[hidden]> writes:
> > I agree.  The combinatorics aren't good, and I even forgot to include the
> > debug/release dimension.  
> 
> Tests aren't generally designed to be run in release mode.  In
> release mode, assertions are turned off.

Whatever. Problems that assert finds show up in release builds anyway, they
are just harder to track down. And since there are issues that only appear in
release, and customers typically get release code, I tend to spend most of my
 test effort getting burn-time in release mode.  
 
> > So the current state of affairs is that there isn't a standard set of flags to
> > control the debug/release and linking options.  
> 
> There certainly is, for debug/release; you stick it in the BUILD
> variable.

That's true, that one is ok.  For some reason it never hit me that all the
current regression tests are run in debug mode -- unless I override it i the
Jamfile ;-)

> > So if the library author wants static linking he basically has to
> > define static linking in the Jamfile.  What I'm thinking is that
> > control of the linking and debug/release options shouldn't have to
> > be specified this way.  
> 
> Why do you think that?  I think your whole view of the static/dynamic
> issue is naive.  

Perhaps.

> Static and dynamic objects can't always be built the
> same way, so that may have to be part of the Jamfile.  

Yes, those rules are typically provided by the library/build/Jamfile.  Why
would I need special rules in the test/Jamfile other than to specify my
dependency on the dynamic or static library?

> And
> static/dynamic linking isn't an all-or-nothing proposition.  Every
> library test involves at least two libraries (the one being tested 
> and the runtime), sometimes more.  There's no inherent reason some 
> couldn't be statically linked and others dynamically linked.

That's irrelevent.  I only care about the library under test.

> Furthermore, I don't see a big advantage in having a separate
> command-line option to choose which of those linking modes is used.

Ok, we disagree on this.

> If the library needs to be tested in several variants, then so be it.
> If it doesn't, but you'd like to see more variants sometimes, you can
> put some of the variants into the --torture option.

Ok.  By the way, I like your suggestion to call it --complete or perhaps
better --exhaustive.

> > The same set of tests should be runnable by the regression tester
> > and hence associated with the column in the results table.  So
> > something like:
> >
> >        VC7.1     VC7.1    VC7.1  etc
> >        release   release  debug
> >        dll       static   dll
> >
> > test1  Pass      fail    Pass
> > test2  Pass      Pass    Pass
> > ...
> >  
> > Then basic/complete option controls the number of tests run.
> 
> That's outta hand, IMO.  If it's worth having options, let's keep
> them simple.

Well, I think it correctly factors the dimensions of compilation options
versus tests.  But I think your earlier email provides an example of how
something similar can be achieved by just using 'if' statements to control the
rules for various linkage options, so I'm fine with baby steps.

Jeff