Re: [boost] Rapid turnaround testing

From: Beman Dawes (bdawes_at_[hidden])
Date: 2007-10-08 09:29:19

Jeff Garland wrote:
> Beman Dawes wrote:
>> As an experiment, I've got a script that is checking svn every 15 
>> minutes, and if there is a change, running the regression tests and 
>> uploading results to a web site.
>>
>> For current VC++ results, see 
>> http://mysite.verizon.net/beman/win32-trunk-results.html
>> On these runs the reporting is set to show a row only if the results are 
>> something other than a simple "pass" without link.
>>
>> For slightly outdated Linux results, see 
>> http://mysite.verizon.net/beman/ubuntu-trunk-results.html
>> Here the results show all tests, regardless of result.
>>
>> The idea is to supplement the regular regression tests at 
>> http://beta.boost.org/development/tests/trunk/developer/serialization.html 
>> with rapid turnaround tests on a few major platforms, aimed at giving 
>> developers quick response as they try to fix bugs affecting platforms 
>> they don't have access to. At the current level failures, the Win32 
>> tests are turning around in 11 minutes while the Linux tests turn in 4 
>> minutes. Thus worst case developers can see test results in 26 minutes!
>>
>> It seems to me this might be helpful to developers. Opinions?
> 
> I think this would be very valuable.
> 
>> Presumably in regular use the upload would go to a centralized site. Or 
>> maybe decentralization is useful for reliability; a boost web page could 
>> contain links. Opinions on that?
> 
> Decentralization is fine with me, but there needs to be something on the 
>   boost site that points to all of them.

Yep.

>> Which reporting do you prefer? My preference is just to show the tests 
>> with something to report.
> 
> I like to see all the tests -- just so that I know they were actually run.

Yes, I worry about that too. Maybe I'll start out showing all results, 
and then reassess later depending on experience.

I've stopped running the tests for now because I'm in the middle of 
switching to a new machine. If all goes well, they will start running 
regularly on Tuesday or Wednesday.

--Beman