RE: [boost] Re: Experimental audience-targeted regression results

From: Aleksey Gurtovoy (agurtovoy_at_[hidden])
Date: 2003-06-19 08:17:52

Vladimir Prus wrote:
> Aleksey Gurtovoy wrote:
> 
> > ... as per 
> http://article.gmane.org/gmane.comp.lib.boost.devel/20648 are
> > available from here:
> > 
> >  * user summary page -
> > http://boost.sourceforge.net/regression-logs/user_summary_page.html
> >  * developer summary page -
> > 
> > http://boost.sourceforge.net/regression-logs/developer_summary
>> _page.html
> >
> > Please comment!
>
> This is very nice! 

Thanks!

> However:
> 1. The developer_result_page.html as well as the user page are rendered
> incorrectly, both in Konqueror and Mozillaa. The first table (for "any")
is
> drawn in the middile of table of content, overlaying it.

Thanks for the report, we will look into it.

> 2. The way results are computed, and how to inperpret them is still not
> clear. For example, I see that "function" is "broken" on borland and
mingw.
> If I were just a user, I'd give up and resort to function pointers. 
> But in fact, mingw only fails a single test, so it's basically usable.
> Borland fails many more tests, but it works good enough still to allow
> program_options library to pass all it's own tests.

Yep. Reporting green in these cases is crucial for the user report to be
fulfill its promise (and it was one of the goals right from the beginning -
http://article.gmane.org/gmane.comp.lib.boost.devel/20648). They _will be_
reported green as soon as somebody with enough knowledge about which tests
are critical and which are not will specify it.

> Probably, library authors should have a way to specify "critical tests".
If
> they don't pass, the library is broken. If some other test fails, it's
> "partially working". 

Yep, that's the idea. Specifying the failures is as simple as putting an
"expected_results.xml" file into the library's "test" directory:

<expected-failures>
    <!-- Example:
    <test-result library="mpl" test-name="is_sequence" toolset="borland"
showstopper="no" />
    <test-result library="mpl" test-name="is_sequence" toolset="cwpro8"
showstopper="no" />
    -->

    </expected-failures>

Thanks for the comments,
Aleksey