[boost] Re: Expected Errors?

From: misha_at_[hidden]
Date: 2003-12-01 07:48:19

David Abrahams <dave_at_[hidden]> writes:

> Misha Bergal <mbergal_at_[hidden]> writes:
>
>> David Abrahams <dave_at_[hidden]> writes:
>>
>>> How do I note that certain tests are expected to fail on a particular
>>> compiler?
>>>
>>
>> To annotate failures in metacomm reports edit file
>> status/explicit-failures-markup.xml. I think the format is
>> self-explanatory. 
>
> OK, thanks!

Looking at the changes you've made to explicit-failures-markup.xml:

1. Cool!
2. You used toolset="*", probably because:

    * it was too painful to replicate the comment to different
       toolsets 

    * you just didn't know which toolsets to specify (Beman and
       we use different toolsets for testing)

    We are aware of these issues. 

    Regarding the first one, it was a conscious decision to do it. I
    didn't have any idea on how often the same comment is applicable
    to different toolsets and didn't want to create something which is
    not goging to be useful. I guess, if there will be more situation
    like yours we will have to add support to more flexible mark
    definition.

    Regarding the second one, I guess some work needs to be done in
    unifying toolsets (toolset names) used for regression testing
    (like http://tinyurl.com/x72s)

> I think it might be helpful for users if there were a color which
> indicates degraded, but still-mostly-working, functionality.  Sort of,
> "this library basically works, but see the failure log for corner
> cases", as opposed to "we expect this to fail and it's been reported
> to the compiler vendor, but it mostly makes the library unusable".
> I'm not sure if we have any of the latter type of failure, so if we
> don't I guess there's not much value in adding a color.  I also
> realize it's a qualitative judgement which you could argue we should
> let users make.  It seems to me, however, that a failure of things
> like is_convertible_fail_test in the iterators library cause only a
> tiny degradation in capability ought to be distinguished from more
> serious problems.

So we need to show the user:

* if the library is completely unusable for particular platform. 

   * may be because of not capable compiler 
   * may be because of it not having been ported to the compiler
   * may be because of it not having been ported to the platform (Mac OS X)

* how significant the particular test (or failure of it) is from the
  library author point of view.

The first needs to be shown on the summary report, the second - on the
detailed report. I guess we need to think about the good visual
representation - I believe that the current visual representation is
simple and powerful and right now I would not trade it's simplicity
for more features.

If we can come up and agree on a good visual representation and unify
the toolsets used for regressions - there are no technical problems
implementing it.

Summary report is easy: if the library is explicitly marked up as
unusable for platform/compiler, then it's status is displayed in
black. 

Detailed report might show the corner case tests with different
background.

I will modify the HTML with current results to reflect the changes I
propose and will post the link to it - it would be easier to discuss
the issue if we have the something to look at.

-- 
Misha Bergal
MetaCommunications Engineering