Re: [Boost-testing] broken reporting of failure expectations

From: David Abrahams (dave_at_[hidden])
Date: 2005-06-29 09:40:39

Aleksey Gurtovoy <agurtovoy_at_[hidden]> writes:

> David Abrahams writes:
>> Aleksey Gurtovoy <agurtovoy_at_[hidden]> writes:
>>
>>>> do you know why it's there?
>>>
>>> I'm afraid you are the only one who can answer that for sure --
>>> http://cvs.sourceforge.net/viewcvs.py/boost/boost/status/explicit-failures-markup.xml?r1=1.1&r2=1.2
>>>
>>> My guess would be that at the time you didn't want to enumerate all
>>> the failing tooselts (most of them, I'd guess), so you've opted for
>>> simply putting a star there and having dark green for the few that
>>> passed (if any).
>>
>> Now I remember what the issue was: I expected that putting the star in
>> there would allow me to set up the message that would be presented
>> *when* there were a failure but that it wouldn't cause the report to
>> say that I expect a failure everywhere.  
>
> Doesn't sound right to me. How do you know whether your comment will
> apply to the new compiler failing the test unless you took a look at
> the failures? 

Because of the nature of the test.  This is a test that certain
special cases will fail to compile.  If they don't fail, you will
normally get a worse error message, later, as the comment indicates:

    This failure is caused by a compiler bug.  Templated operators
    that combine different iterators built with iterator_facade or
    iterator_adaptor may be present in an overload set even when those
    iterators are not interoperable.  The usual result is that error
    messages generated by illegal use of these operators will be of
    lower quality.

> And why should it be marked up as expected by default, anyway?

It shouldn't; that's exactly my point.  What should happen is that
there should be a default comment for when the test fails, but that
the test should be expected to succeed.

>> When I expect a failure
>> everywhere, I use an expected-fail test (e.g. compile-fail). 
>
> I'm afraid you are confusing explicit markup with test-case types.

No, I'm not.  I claim the very availability of '*' in the explicit
markup *with its current meaning* confuses explicit markup with test
case types.

> The markup serves a purpose of accounting for compiler bugs we don't
> want to/know how to workaround, while test cases assert the ideal
> expected behavior. They are orthogonal.

I know that.

>> Is there really any use for the current behavior?
>
> Yes. The fact that you could mark a test case as an expected failure
> for all toolsets at once is a red herring, really. 

No, it's the central question I'm asking.  Why would one ever want to
do that?

> Whether the toolset name contains the star or a name of the
> particular toolset, the markup behavior is the same.

I'm arguing for different behavior in this case.

-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com