Re: [Boost-testing] broken reporting of failure expectations

From: David Abrahams (dave_at_[hidden])
Date: 2005-07-03 22:55:01

Aleksey Gurtovoy <agurtovoy_at_[hidden]> writes:

> David Abrahams writes:
>> I'm afraid you are confusing failure to compile with failure to pass
>> the test ;-)
>
> Don't you think that people who brought you the current regression
> reports know better than that? ;)
>
>>
>> Compilation is expected to fail.  If that happens, I want a green
>> square.  If compilation succeeds (which is a failure of the test
>> case), I want the comment I quoted in my previous message.
>>
>>>> I claim the very availability of '*' in the explicit markup *with
>>>> its current meaning* confuses explicit markup with test case types.
>>>
>>> ? _You_ might be confused; the markup doesn't confuse anything.  It
>>> has a totally consistent semantics which keeps orthogonal things,
>>> well, orthogonal.
>>
>> It may be consistent, but it's redundant and gives useless results. 
>
> I don't necessarily disagree with this.
>
>> A "compile-fail" test marked with a "*" using the current semantics
>> should just be changed to a "compile" test.  A "compile" test marked
>> with a "*" using the current semantics should be changed to a
>> "compile-fail" test.
>
> These are not equivalent, 

What are not equivalent?  If you're saying that a "compile-fail" test
marked with a "*" using the current semantics is not equivalent to a
"compile" test, well obviously they get reported differently.  But the
way the former is reported is pretty useless.

> and I don't see how doing so would help anything or anybody.

Doing what?  When I said "should just be changed" I meant that anyone
who might be tempted to write a "*" toolset markup should instead
change the test expectation in his Jamfile.

>> IIRC I only used that syntax in the first place after asking how to
>> support that use case and getting told that "*" would do it.  Maybe I
>> misunderstood Misha's responses in
>> http://news.gmane.org/find-root.php?message_id=%3c20031201120725.C86570%40monica.cs.rpi.edu%3e
>
> I'm afriad so.
>
>> but I assumed when he said "we will have to add support to more
>> flexible mark definition" he meant that this was going to be OK.
>
> I'm afraid it wasn't clear to us from the cited thread that you've
> actually expected something different from what we had in mind (==
> the current behavior).

I'm pretty sure the current behavior was already in place, and Misha
clearly acknowledged that something else was needed in that thread.

>> I don't know if there's any point in arguing about how much of a
>> burden it will be.  I was under the impression that this was a usage
>> that would be supported.  If it isn't going to be supported, I'll
>> write out the explicit entries. 
>
> Well, I was trying to make sure I understand your use cases completely
> so that together we can make an informed decision whether it's worth
> supporting or not. It's definitely a newly discovered use case for us,
> and it would cost _something_ to implement, but if we decide it's
> worth it, at the very least we can put it on the TODO list. Yes, so
> far it doesn't look worthy the trouble.

We can see how often I have to change those comments and decide later.
Probably the bigger problem (not that it's major) is that there's no
way to capture the real intended semantics of the comment in the xml
file.

>> Just let me know which.
>
> Whatever the consensus is going to be, in the short term explicit
> entries are the way to go.

Okay, I'll fix those up.

-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com