|
|
Boost Testing : |
From: Beman Dawes (bdawes_at_[hidden])
Date: 2007-11-15 22:07:58
David Abrahams wrote:
> on Thu Nov 15 2007, Beman Dawes <bdawes-AT-acm.org> wrote:
>
>> David Abrahams wrote:
>>
>>> We've removed the BittenToDo page and entered the things we could
>>> think of to do as tickets. See
>>> https://boost-consulting.com/trac/projects/boost/query?status=accepted&status=assigned&status=new&status=reopened&order=priority&col=id&col=summary&col=status&col=type&col=priority&col=milestone&col=component&row=description
>> #8, Automatic ticket creation, is questionable in my mind.
>>
>> What often happens is that there is some testing snafu that causes a
>> huge number of tests to fail. I'd hate to see a tracker littered with
>> tickets for each of the failed tests.
>
> Obviously we'd have to be smarter about it. For example, if more than
> 10% of a library's tests are failing we should create one ticket, and
> if a platform is failing (nearly) everything, we should create a
> ticket for the machine owner.
Yep. Another heuristic would be to create a ticket for the machine owner
if more than n tests are missing. n is probably pretty small, too. Maybe
2 or 3.
>> #4, Show output for failed tests, and #2, Support expected failures
>> markup, are the high priority issues. I'm sure you and Daniel
>> already knew that:-)
>
> Yes. We have some strong feelings that we're not handling expected
> failure markup in the best way right now, so "support" at least
> initially will probably look a little different than it does now.
>
>> I find the hand editing of the markup file tedious at best and error
>> prone at worst. Just dreaming, bu I'd rather go to a web page and
>> fill out a form.
>
> I suppose you didn't read all of
> https://boost-consulting.com/trac/projects/boost/ticket/9
> :)
Ah! I missed the last sentence. You are ahead of me:-)
>
>> I hope that work on Bitten won't slow down work on eliminating
>> process_jam_log.
>
> Well, the fact that I have a system working that uses process_jam_log
> right now makes eliminating it a lower priority item for me than some
> other things that are crucial to having the system actually be useful.
>
>> I've again wasted several hours today because
>> Boost.Build doesn't generate XML files directly.
>
> How so?
Setting up new release branch test runners, and process_jam_log refusing
to build.
I'd guess there has been at least one issue a week with process_jam_log
for the four or five weeks I've been working on 1.35.0 release issues.
> My problem is that I don't really understand what's involved in
> eliminating PJL yet. Rene has referred to __ACTION_RULE__ but I don't
> see how to use that facility to write any files.
Maybe just have the compile, link, and run commands redirect stdout and
stderr to a file (or possibly one file for each kind of action)?
In other words, there really isn't an absolute requirement that the bjam
step put everything in one XML file per test. As long as the location
and name of output capture files were standard (maybe the names of the
output capture files were written into the XML files), a post-bjam step
could package them up at the same time the XML files are packaged for
transmission.
--Beman