Re: [Boost-testing] broken reporting of failure expectations

From: Aleksey Gurtovoy (agurtovoy_at_[hidden])
Date: 2005-07-01 01:51:15

David Abrahams writes:
> I'm afraid you are confusing failure to compile with failure to pass
> the test ;-)

Don't you think that people who brought you the current regression
reports know better than that? ;)

>
> Compilation is expected to fail.  If that happens, I want a green
> square.  If compilation succeeds (which is a failure of the test
> case), I want the comment I quoted in my previous message.
>
>>> I claim the very availability of '*' in the explicit markup *with
>>> its current meaning* confuses explicit markup with test case types.
>>
>> ? _You_ might be confused; the markup doesn't confuse anything.  It
>> has a totally consistent semantics which keeps orthogonal things,
>> well, orthogonal.
>
> It may be consistent, but it's redundant and gives useless results. 

I don't necessarily disagree with this.

> A "compile-fail" test marked with a "*" using the current semantics
> should just be changed to a "compile" test.  A "compile" test marked
> with a "*" using the current semantics should be changed to a
> "compile-fail" test.

These are not equivalent, and I don't see how doing so would help
anything or anybody.

[...]

>
> IIRC I only used that syntax in the first place after asking how to
> support that use case and getting told that "*" would do it.  Maybe I
> misunderstood Misha's responses in
> http://news.gmane.org/find-root.php?message_id=%3c20031201120725.C86570%40monica.cs.rpi.edu%3e

I'm afriad so.

> but I assumed when he said "we will have to add support to more
> flexible mark definition" he meant that this was going to be OK.

I'm afraid it wasn't clear to us from the cited thread that you've
actually expected something different from what we had in mind (== the
current behavior).

>
>> given that the resulting semantics would be quite different from other
>> cases which are going to look exactly the same to a newcomer.
>>
>> I'm not even sure the use case is worth supporting: granted,
>> currently you have to update the list of toolsets every time a new
>> broken compiler appears in the reports
>
> Or the same broken compiler appears with a new toolset name, which was
> happening a lot when I made that entry.

OK.

>
>> and in this particular case you could have avoided it since you know
>> that the comment is going to be the same regardless any toolset
>> details, but is it really that much of a burden, considering that
>> you still have to do a similar (or larger) maintenance work for
>> other test cases that are not that narrow?
>
> I don't know if there's any point in arguing about how much of a
> burden it will be.  I was under the impression that this was a usage
> that would be supported.  If it isn't going to be supported, I'll
> write out the explicit entries. 

Well, I was trying to make sure I understand your use cases completely
so that together we can make an informed decision whether it's worth
supporting or not. It's definitely a newly discovered use case for us,
and it would cost _something_ to implement, but if we decide it's
worth it, at the very least we can put it on the TODO list. Yes, so
far it doesn't look worthy the trouble.

> Just let me know which.

Whatever the consensus is going to be, in the short term explicit
entries are the way to go.

-- 
Aleksey Gurtovoy
MetaCommunications Engineering