Re: [boost] Testing Farm

From: Jonathan Wakely (cow_at_[hidden])
Date: 2005-03-21 13:16:48

On Mon, Mar 21, 2005 at 05:37:07PM +0100, Martin Wille wrote:

> Jonathan Wakely wrote:
> >On Sun, Mar 20, 2005 at 04:52:23PM -0700, Jeff Garland wrote:
> >
> >
> >>Seems like the real variation here is in OS's. Of course, there are 
> >>hundreds
> >>of Linux distros so Linux isn't just Linux -- not sure which one to pick
> >
> >
> >But there's only one glibc, which they all use.  Different glibc versions
> >shouldn't be _too_ different, for Boost's purposes.
> 
> 
> In my experience, they make a difference. And there's not only one 
> glibc, since there are also many versions of it. Tests also depend on 
> the kernel version, on the CPU(s), on the version of the system 
> compiler, on the Python version et cetera.

Then I stand corrected.

(The system compiler surely only matters if you're using the system
compiler?  If you do parallel installs of five versions of GCC and use
them for the tests, does the system compiler matter?)

I may well be wrong again, but IMHO aiming for wider coverage (by
testing on a couple of versions of Linux and some version of FreeBSD,
for example) would pick up more portability problems than testing on a
number of subtley different flavours of Linux.

I know I've found several problems when I run the tests on FreeBSD that
have clearly never showed up on any of the versions of Linux that are
tested by other people.

Even if it does make quite a difference, it seems obvious to choose from
the most popular distros, since that ensures compatibility with a higher
number of users.  Choosing to run tests on the more obscure distros
might find quirks of those distros but it might be that no Boost user
ever uses that distro.

>   A distro that doesn't ship with Boost might make
> >sense, since presumably the distros that provide precompiled versions of
> >Boost are already testing it.
> 
> I doubt any distributor runs the regression tests.

Again, I'm a bit surprised.

> In fact, support for Boost packages on various Linux distributions isn't 
> too great. There's no reason to expect distributors to run the tests.
> Also, it actually doesn't matter whether a distribution tested the Boost 
> version it has a package for. It's an already released version which 
> might even contain patches from the distributor. We're talking about 
> testing from the CVS, though.

If *I* was the maintainer of a distro's Boost package I'd test against
CVS, to know what problems might be coming up soon, but I'm not, so my
opinion counts for nowt!

jon

-- 
"Once, during Prohibition, I was forced to live for days on nothing but 
 food and water."
	- W.C. Fields