Re: [boost] Re: [Boost.Test] New testing procedure?

From: Jeff Garland (jeff_at_[hidden])
Date: 2004-05-22 18:18:31

On Sat, 22 May 2004 17:18:04 -0500, Aleksey Gurtovoy wrote
 
> > Anyway, I remember others (Beman) have
> > previously expressed concern about the length of the test cycle.
> 
> It is a problem if you are running them on your "primary" machine 
> during the day. I don't think we can do much about it -- just 
> compiling the tests takes about half of the whole cycle's time, and 
> personally I see little value in regressions that at least didn't 
> compile every test.

Well, I think there is.  The additional value of compiling and running the
exact same test for date_time in the dll and static link version is exactly
the sort of thing that could reduce the compile and runtimes for 'primary
machine' testers.  I suppose I could start customizing my Jamfile to only run
multi-threaded dll on windows, but now I'm deciding what the regression
testers can afford and it stops you (Meta-Comm) from running all the
variations -- which I still want to see.

> > > > Jeff wrote: 
> > > > Serialization tests alone dramatically increase the
> > > > length of the time to run the regression if we always run the 
> > full test.
> ...snip...
> Sure, I was just saying that the library author can deal with it on its
> own -- just make several sections in the bjam file and 
> enable/disable them depending on your current needs.

Sure, but really I'm proposing we turn that around.  If the regression tester
 has the hardware resources to run a torture test with 3 different linking
variations then they should be able to do that.  As soon as Robert enables the
2.5 hour torture test regression testers might suddenly have an objection to
'author only' control.
 
> > Perhaps. From my view things seem pretty thin already. 
> 
> If we provide a documented way to setup the whole thing, and post "A 
> Call for Regression Runners", I am sure we'll get some response.
> 
> > There was some
> > discussion during the last release that some testers had removed the python
> > tests because they were taking too long.
> 
> Well, you are right that right now the resources are a little spare, 
> but IMO it's just because we didn't work on it.

You could be right.
 
> > BTW, just to pile on, wouldn't it be nice if we had testing of the sandbox
> > libraries as well?  This would really help those new libraries get ported
> > sooner rather than later...
> 
> IMO that's asking too much. Many of them never get submitted.

Many are extensions to existing libraries under active development -- likely
to get moved to final CVS.  I think it would be nice for libraries coming up
for review to get the benefit of the regression system.  Clearly we might need
to subset what gets run and clean out the old stuff, but I think this would
smooth the integration of new libraries.

> "Basic" (supposedly what we have now) versus "drastic" (supposedly 
> what's coming with serialization) distinction definitely makes 
> sense. I am not arguing against this one, rather against lowering 
> down our current standards.

I don't want to lower the current standard either.  With the Basic option,
however, some current libraries might define a smaller test suite speeding up
the core tests.  Of course, if there it is impossible to subset, then fine
they could stay where they are now.  Those regression sites that have the
horsepower to run the torture test with all variations can still go for that
option.  Of course we will prefer that, but some might choose to run the
torture test once per week (say over a weekend) and the regular tests during
the week.
 
> > If the test takes 5 to 6
> > hours to run a single compiler we might lose the one contributor we have.
> 
> True, if they are forced to run the drastic test, which IMO 
> shouldn't be the case -- it should be entirely up to the regression 
> runner to decide when and if they have the resources to do that.

Well as soon as Robert wants to run the torture test he's going to get it at
all sites if he controls it via his Jamfile.  So we need some boost-wide
option to define these variations.  Hopefully my other email clarifies the idea.
 
> >  For awhile I'm likely to setup only a single compiler (gcc 3.3.1) on my
> > Mandrake 9 machine.  With that approach I should be able to cycle more
> > frequently.  Incremental testing is probably a good thing to try out as
> > well.
> 
> It produces less reliable results, but the roots of it needs to be 
> tracked and fixed, so yes, it would be good to start looking into it.

Ok will do...

Jeff