[boost] Re: Expected Errors?

From: David Abrahams (dave_at_[hidden])
Date: 2003-12-01 09:57:02

misha_at_[hidden] writes:

> David Abrahams <dave_at_[hidden]> writes:
>
>> Misha Bergal <mbergal_at_[hidden]> writes:
>>
>>> David Abrahams <dave_at_[hidden]> writes:
>>>
>>>> How do I note that certain tests are expected to fail on a particular
>>>> compiler?
>>>>
>>>
>>> To annotate failures in metacomm reports edit file
>>> status/explicit-failures-markup.xml. I think the format is
>>> self-explanatory. 
>>
>> OK, thanks!
>
> Looking at the changes you've made to explicit-failures-markup.xml:
>
> 1. Cool!
> 2. You used toolset="*", probably because:
>
>     * it was too painful to replicate the comment to different
>        toolsets 
>
>     * you just didn't know which toolsets to specify (Beman and
>        we use different toolsets for testing)

No, it's because those tests failing is indicative of unimplimented
SFINAE.  When they fail, the same comment really does apply in all
cases.

>     We are aware of these issues. 
>
>     Regarding the first one, it was a conscious decision to do it. 

Sorry, to "do what"?  Make it painful to replicate the comment?

>     I didn't have any idea on how often the same comment is
>     applicable to different toolsets and didn't want to create
>     something which is not goging to be useful. I guess, if there
>     will be more situation like yours we will have to add support to
>     more flexible mark definition.

?? I thought tollset="*" is supported already.  Last night's tables
seem to reflect that belief.

>     Regarding the second one, I guess some work needs to be done in
>     unifying toolsets (toolset names) used for regression testing
>     (like http://tinyurl.com/x72s)

Probably a good idea.

>> I think it might be helpful for users if there were a color which
>> indicates degraded, but still-mostly-working, functionality.  Sort of,
>> "this library basically works, but see the failure log for corner
>> cases", as opposed to "we expect this to fail and it's been reported
>> to the compiler vendor, but it mostly makes the library unusable".
>> I'm not sure if we have any of the latter type of failure, so if we
>> don't I guess there's not much value in adding a color.  I also
>> realize it's a qualitative judgement which you could argue we should
>> let users make.  It seems to me, however, that a failure of things
>> like is_convertible_fail_test in the iterators library cause only a
>> tiny degradation in capability ought to be distinguished from more
>> serious problems.
>
> So we need to show the user:
>
> * if the library is completely unusable for particular platform. 
>
>    * may be because of not capable compiler 
>    * may be because of it not having been ported to the compiler
>    * may be because of it not having been ported to the platform (Mac OS X)
>
> * how significant the particular test (or failure of it) is from the
>   library author point of view.

Yeah.  We have a number of less-significant tests for corner cases in
the type traits library, for example.

> The first needs to be shown on the summary report, the second - on the
> detailed report. I guess we need to think about the good visual
> representation - I believe that the current visual representation is
> simple and powerful and right now I would not trade it's simplicity
> for more features.

Agreed.

> If we can come up and agree on a good visual representation and unify
> the toolsets used for regressions - there are no technical problems
> implementing it.
>
> Summary report is easy: if the library is explicitly marked up as
> unusable for platform/compiler, then it's status is displayed in
> black. 
>
> Detailed report might show the corner case tests with different
> background.
>
> I will modify the HTML with current results to reflect the changes I
> propose and will post the link to it - it would be easier to discuss
> the issue if we have the something to look at.

Great!

-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com