RE: [boost] Experimental audience-targeted regression results

From: Aleksey Gurtovoy (agurtovoy_at_[hidden])
Date: 2003-06-19 08:00:24

Peter Dimov wrote:
> Aleksey Gurtovoy wrote:
> > ... as per http://article.gmane.org/gmane.comp.lib.boost.devel/20648
> > are available from here:
> >
> >  * user summary page -
> > http://boost.sourceforge.net/regression-logs/user_summary_page.html
> >  * developer summary page -
> > 
> http://boost.sourceforge.net/regression-logs/developer_summary
> _page.html
> >
> > Please comment!
> 
> Your como-win32 installation is still broken, it seems.

Yeah, kind of. Since it's unfair to report any results under these
conditions, and we don't have resources to solve the issue at the moment,
for the time being Comeau C++ is excluded from the experimental run.

> The summaries are nice, but the red "broken" thing on the user page may be
> too intimidating, 

When will show the actual status, it shouldn't be (it doesn't yet, since
some cooperation from library authors is needed - please see below). Or,
rather, if it still is, then it's the status that is too intimidating, not
the report :). 

> esp. in cases where the library actually works
> (everything/como-win32, smart_ptr/intel-win32) but the tests fail for
> various obscure reasons (setup issues, nonstandard auto_ptr).

I wholeheartedly agree it's important to show green when the library
actually works. I was the one who argued for it in the first place :) - once
again, see http://article.gmane.org/gmane.comp.lib.boost.devel/20648.

If some failures are not really indicative of the library's status, the user
summary report will happily show you a green "OK*" thingy - given that the
library author specified that these tests are expected to fail for the
particular toolset and that the failures are non-critical. The paragraph at
the top of
http://boost.sourceforge.net/regression-logs/user_result_page.html tries to
explain how to do that. Specifying the expected failures will also clear the
corresponding reds from the developer pages, bringing it closer to the state
when one can use the summary report to monitor new regressions.

As for the setup issues, when you have those, you shouldn't publish the
results until you solve them. We are guilty we did.

Thanks for the comments,
Aleksey