Re: [boost] Re: [Boost.Test] New testing procedure?

From: Aleksey Gurtovoy (agurtovoy_at_[hidden])
Date: 2004-05-22 15:04:19

Jeff Garland writes:
> On Fri, 21 May 2004 20:41:59 -0400, David Abrahams wrote
> > "Robert Ramey" <ramey_at_[hidden]> writes:
> > 
> > > Dave Abrahams wrote:
> > > I believe we will have to move to an "on demand" model for most testing
> > > while reserving "total coverage" testing for just prior to release.  
> > 
> > I don't.  You can get test results for any library on any compiler
> > that's being tested daily within 24 hours.  Some compilers are tested
> > every 12 hours (see meta-comm).  I don't see why that should be 
> > insufficient.  
> 
> It's kind of spotty outside of the meta-comm guys:
> IBM Aix 11 days
> Mac OS  today
> SGI Irix 2 weeks
> linux 4 days
> Sun Solaris 6 days
> Win32 4 weeks
> win32_metacomm today
> 
> And that's today.

IMO the only thing it indicates is that these tests are initiated manually.

> Consider during the next couple months 3-4 new libraries
> are pending to be added.  

No a problem, in general. Right now a *full rebuild* takes about 8 hours.
If we switch to incremental model, we have plenty of reserve, here.

> Serialization tests alone dramatically increase the
> length of the time to run the regression if we always run the full test.

Well, the dramatic cases need to be dealt with, and IMO a Jamfile that 
allows the library author to manage the level of "stressfulness" would be 
just enough.

> What will happen in a year when we have say 10 new libraries?

Well, hopefully we'll also have more computing power. Surely a lot of 
organizations which use Boost libraries can afford to spare a middle-class 
machine for automatic testing?

>  
> Robert and I have believe something will need to be done.  We've tried to
> start a discussion, but no one responded:
> 
> http://lists.boost.org/MailArchives/boost/msg64471.php
> http://lists.boost.org/MailArchives/boost/msg64491.php
> 
> Jeff
> 
> BTW I might be able to contribute to the Linux testing -- are there
> instructions on how to set this up somewhere?

For *nix systems, there is a shell script that is pretty much self-explanatory:

http://cvs.sourceforge.net/viewcvs.py/boost/boost/tools/regression/run_tests.sh

If you want something that requires even less maintenance, we can provide 
you with the Python-based regression system we use here at Meta.

--
Aleksey Gurtovoy
MetaCommunications Engineering