Re: [boost] Re: [Boost.Test] New testing procedure?

From: Aleksey Gurtovoy (agurtovoy_at_[hidden])
Date: 2004-05-22 22:21:29

Jeff Garland writes:
> > Sure, I was just saying that the library author can deal with it on its
> > own -- just make several sections in the bjam file and 
> > enable/disable them depending on your current needs.
> 
> Sure, but really I'm proposing we turn that around.  If the regression tester
>  has the hardware resources to run a torture test with 3 different linking
> variations then they should be able to do that.  As soon as Robert enables the
> 2.5 hour torture test regression testers might suddenly have an objection to
> 'author only' control.

You have a point, here.

> > > BTW, just to pile on, wouldn't it be nice if we had testing of the sandbox
> > > libraries as well?  This would really help those new libraries get ported
> > > sooner rather than later...
> > 
> > IMO that's asking too much. Many of them never get submitted.
> 
> Many are extensions to existing libraries under active development -- likely
> to get moved to final CVS.  I think it would be nice for libraries coming up
> for review to get the benefit of the regression system.  Clearly we might need
> to subset what gets run and clean out the old stuff, but I think this would
> smooth the integration of new libraries.

I agree it would be nice, but that's basically it. If we happened to have the 
resources to do that, good, if we don't, well, we don't. 

> 
> > "Basic" (supposedly what we have now) versus "drastic" (supposedly 
> > what's coming with serialization) distinction definitely makes 
> > sense. I am not arguing against this one, rather against lowering 
> > down our current standards.
> 
> I don't want to lower the current standard either.  With the Basic option,
> however, some current libraries might define a smaller test suite speeding up
> the core tests.  Of course, if there it is impossible to subset, then fine
> they could stay where they are now.  Those regression sites that have the
> horsepower to run the torture test with all variations can still go for that
> option.  Of course we will prefer that, but some might choose to run the
> torture test once per week (say over a weekend) and the regular tests during
> the week.

I think I'm becoming comfortable with the idea. It does complicate things, 
however. For instance, now the library author would have to look into twice
as much reports to determine the state of her library (superseding the old 
"torture" results with the new "basic" ones would make the whole thing
useless).

>  
> > > If the test takes 5 to 6
> > > hours to run a single compiler we might lose the one contributor we have.
> > 
> > True, if they are forced to run the drastic test, which IMO 
> > shouldn't be the case -- it should be entirely up to the regression 
> > runner to decide when and if they have the resources to do that.
> 
> Well as soon as Robert wants to run the torture test he's going to get it at
> all sites if he controls it via his Jamfile.  So we need some boost-wide
> option to define these variations.  Hopefully my other email clarifies the idea.

It does, although I don't see how we can manage/afford all the combinations.

--
Aleksey Gurtovoy
MetaCommunications Engineering