|
|
Boost Testing : |
From: Misha Bergal (mbergal_at_[hidden])
Date: 2005-03-09 15:54:34
Rene Rivera <grafik.list_at_[hidden]> writes:
--text follows this line--
Rene Rivera <grafik.list_at_[hidden]> writes:
> All,
>
> I've been reading all the testing related post and not having time to
> respond to them. So I decided to talk about what might be done to
> improve things in one big posting.
>
> For along time now one of my objectives has been to use BuildBot
> (http://buildbot.sf.net/) to improve the management of running the
> regression tests. For those who don't feel like reading about that
> software here's a quick summary:
>
Several questions:
1. You presume that when BuildBot tells certain machine to do
something it is already done with last changes it needed to
process. It is not so now. I believe our cycle is ~11 hours and
Martin's is ~24 hours.
2. I still don't understand:
* What exactly will BuildBot tell slaves to build.
* What BuildBot decides they need to build. What clients
declare they can build?
* What would be the display result format. Green signifies what?
Red signifies what? Pointing to some already running BuilBot
would be helpful. Looking at http://buildbot.ethereal.com/ I just
don't see how the library author would know if she has broken
something.
[...]
> *Resources*
> The only predictable way to address the resource usage, is to
> distribute the testing so we can create more capacity.
Or don't waste what we already have, see below.
[...]
> *Response*
> The gain from segmentation and distribution of testing is hopefully
> obvious ;-) But another advantage of using BuildBot is that we are not
> tied to waiting for the XSLT processing to see results. Sure the
> results are not going to be as incredibly well organized as the
> Meta-Comm results but they are immediately available.
I can see how looking at the logs would be helpful, if the build was
done per CVS commit. I commit, get a change number and look how that
was processed by all slaves. Unfortunately because of build times it
is not possible now - see below.
> *Releases*
> Managing the testing for a release was brought up a many times. And
> it's clear that requiring testers to do manual changes is just not
> working.
What are the specific use cases you are referring to?
[...]
>From my point of view, this is what needs to be worked on next
(http://www.crystalclearsoftware.com/cgi-bin/boost_wiki/wiki.pl?Boost.Testing)
# Incremental testing is not reliable
# Tests are run for compilers for which they are known to fail.
This needs to be done no matter whether we use BuildBot or something
else.
-- Misha Bergal MetaCommunications Engineering