Re: [boost] Understanding the test matrix

Subject: Re: [boost] Understanding the test matrix
From: Jim Bell (Jim_at_[hidden])
Date: 2010-10-20 09:59:50

> The release test matrix seems to be telling me that there are many failures
> on different platforms, and in fact
> that there is no platform with no failures on all libraries, or libraries
> with no failures on all platforms. We just
> live with that, is that right?
>
> Do all the failures get fixed before an actual release is made, or noted in
> the release notes?

I think the original philosophy was that it should work on mainstream,
modern compilers/platforms, but not necessarily on your old/outlier, and
here's a matrix of what's been tested. I'm okay with that.

Also, a single unexpected regression failure marks the whole library as
fail/yellow on the main matrix. And something like serialization not
linking cascades to several other libraries showing failed (e.g.,
date_time on mingw32). It should be that way, but it shakes the casual
observer's confidence in all of boost.

All this to say, maximizing the green on that regression matrix is worth
the work, and worth holding up a release, IMHO.

Also, many fixes are not as simple as just looking at the test results
for a compiler/platform you have little experience with and can't test
first-hand. (E.g., Ticket # 4736 -- it's clear the author was looking
for the failure, but the only thing that made progress was my running it
in gdb on that particular platform.)