Re: [boost] Re: [Boost.Test] New testing procedure?

From: Jeff Garland (jeff_at_[hidden])
Date: 2004-05-23 00:17:39

On Sat, 22 May 2004 22:21:29 -0500, Aleksey Gurtovoy wrote
> > > > wouldn't it be nice if we had testing of the sandbox
> > > > libraries as well? 
>
> I agree it would be nice, but that's basically it. If we happened to 
> have the resources to do that, good, if we don't, well, we don't. 

I agree, it's a nice to have.  I was just trying to make the point that there
is more that could be done...
 
> I think I'm becoming comfortable with the idea. It does complicate 
> things, however. For instance, now the library author would have to 
> look into twice as much reports to determine the state of her 
> library (superseding the old "torture" results with the new "basic" 
> ones would make the whole thing useless).

Yeah, I agree that's an issue.  Although I expect the basic results to
normally be a subset of the torture results.  So they could be overlayed, but
then some of the results are out of date.  Probably better would be to have
totally different pages for the basic and torture results.  Which of course
makes it harder to find the results.  On the other hand, we might be wringing
our hands about problems that aren't important.  Maybe Meta-Comm is always
going to run the torture test in incremental mode while someone else just runs
basic tests normally.  Then before a release we could ask the testers that can
to notch up to the complete test.

> > Well as soon as Robert wants to run the torture test he's going to get 
> > it at all sites if he controls it via his Jamfile.  So we need some
> > boost-wide option to define these variations.  Hopefully my other 
> > email clarifies the idea.
> 
> It does, although I don't see how we can manage/afford all the combinations.

I agree.  The combinatorics aren't good, and I even forgot to include the
debug/release dimension.  But this is where additional testers could help. 
One group could run with release builds while another runs debug builds.  

So the current state of affairs is that there isn't a standard set of flags to
control the debug/release and linking options.  So if the library author wants
static linking he basically has to define static linking in the Jamfile.  What
I'm thinking is that control of the linking and debug/release options
shouldn't have to be specified this way.  The same set of tests should be
runnable by the regression tester and hence associated with the column in the
results table.  So something like:

       VC7.1     VC7.1    VC7.1  etc
       release   release  debug
       dll       static   dll

test1  Pass      fail    Pass
test2  Pass      Pass    Pass
...
 
Then basic/complete option controls the number of tests run.

Jeff