Re: [boost] Re: Regression report for Releases?

From: Rene Rivera (grafik.list_at_[hidden])
Date: 2005-03-21 21:47:56

Jeff Garland wrote:
> On Sun, 20 Mar 2005 13:53:48 -0600, Rene Rivera wrote
> 
>>Jeff Garland wrote:
>>
>>
>>>Do we even have a way of tracking the check-ins?  That might be a 
>>>good first
>>>step.  I notice that sourceforge seems to be generating some sort of email
>>>when I check-in, but I don't know of a way to subscribe to the changelist. 
>>
>>Yes we do. Dave and I, long ago, set up those emails SF sends so we 
>>could get Buildbot to work. So if we track CVS changes to the 
>>individual builds we can tell who and what breaks a build. Even 
>>though I'm still working on the buildbot here's a sneak peek.. 
>>http://build.redshift-software.com:9990/
> 
> 
> Very cool!  No doubt buildbot will be a great asset -- when do you think it
> will be ready to 'go production'?  

My goal is to have it doing Linux regressions (gcc-release) by next 
week. I taking it carefully, and hence slowly, as it's crucial to reduce 
  the chances of any test system from breaking. So I do some changes and 
let the thing run for a day to see if anything strange happens.

After it's running on my limited setup we can talk about expanding to 
other brave testers out there :-)

>>>others run several.  So depending on when you check-in, it takes up to a
>>>couple days to really see the results of all platforms/compilers. 
>>>The only way I see us getting closer to the ideal is more machines 
>>>really dedicated to just Boost testing...
>>
>>Or going to an active system like Buildbot.
> 
> I don't think the existence of Buildbot solves all of our resource issues. 

Nothing can solve every problem, unfortunately.

> I
> would expect only a limited number of the current regression testers will be
> able to install and use Buildbot -- I'm certain there will be firewall and
> other issues for some that just stop this from happening. 

Proxies can solve most firewall problems, so I wouldn't worry too much 
about that. As for the requirements of running Buildbot itself, they are 
equivalent to those of using the current regression.py script. But yes 
there will be issues just getting the setup working.

> Plus if it takes 5
> hours to run a Boost build you will still have a long delay before you find
> out if something is broken. 

At minimum with Buildbot you can see the build log live. So if the build 
you triggered only builds a small part of the overall Boost then you'll 
get to see results almost immediately. Obviously I'm making two 
assumptions here.. that we can resolve some of the incremental testing 
problems, and that your changes don't cause a Boost wide rebuild like 
changing something like type traits would.

> For most developers they would like to see a
> library focused rebuild, which for most could happen in minutes.  As an
> example, since almost nothing in Boost depends on date-time it's very hard for
> me to break all of Boost.  So rerunning all of the Boost regression for a
> date-time check-in is mostly a waste of resources.

Definitely. I made the suggestion earlier that we should break up the 
testing so that some testers can devote resources to only testing 
subsets of Boost:

http://permalink.gmane.org/gmane.comp.lib.boost.testing/392

(I know it's a long post.. The scalability section is what I'm referring 
to.)

-- 
-- Grafik - Don't Assume Anything
-- Redshift Software, Inc. - http://redshift-software.com
-- rrivera/acm.org - grafik/redshift-software.com - 102708583/icq