Boost logo

Boost :

Subject: Re: [boost] [test] Patches to improve test log detail for better thirdparty integration
From: Jamie Allsop (ja11sop_at_[hidden])
Date: 2012-09-26 08:17:14


On 25/09/12 18:14, Gennadiy Rozental wrote:
> Jamie Allsop<ja11sop<at> yahoo.co.uk> writes:
>
>>
>> These are patches we've been using for some time locally, the recent
>> discussions highlighted the importance of pushing patches like these to
>> a wider audience.
>>
>> In summary the key changes are:
>>
>> * https://svn.boost.org/trac/boost/ticket/7410
>> * https://svn.boost.org/trac/boost/ticket/7397
>
> These are fine. I'll try to check it in on a weekend

Excellent thanks!

>
>> * https://svn.boost.org/trac/boost/ticket/7417
>
> This is not a good direction IMO. This breaks clear separation between test log
> and report streams. We also do not want duplication of information in two
> streams. We should consider an approach where report stream is being produced on
> a fly (in parallel) during test execution. This might be a bit tricky and is not
> always desirable, but can be enables by special command line switch.

Interesting idea - having the report be produced on the fly might work
well, especially if using human readable format. Essentially you are
only a couple of regexes away from useful output. I suppose my main need
is to be able to see test progress as:

test XXX started...
<test now progressing - could take some time>
test XXX completed - status - elapsed - assertions passed out of total

Having said all that I do not see the harm is adding this to the test
log. Here's why. The test log already specifies when a test is started
and when a test completes. When a test completes it already gives you
the elapsed time. Often you *want* to see the test log output with other
messages and so on that you write to the file. Moreover you tend to
parse or interpret the test log on the file, tail it or whatever.
Therefore simply adding the test status and assertion counts in addition
to the time elapsed seems like a good fit to me.

I see the role of the report stream as being very different. In the
report stream I see the desire to create an arbitrary report in an
arbitrary format (like XML). The information contained in the report
stream or the log stream is not really the issue.

In the log stream you get the logged messages *and* the basic test
detail as a simple string so as a human you can read the log and know
that test XXX started 'here' output these message and then completed
after a period of time. Knowing also whether test XXX passed or failed
after that time period and how many assertions were successfully
executed is only a slight addition to the current detail. The key point
is the log stream is often there to be 'read'.

The report stream contains essentially the same test information as the
log stream then, except that it *only* contains the test status detail.
In essence the report is more like a summary (no log messages and so on
- just the test detail). It's primary function is to create a report
that can then be used for later interpretation (often by a CI system) -
not really for 'reading'.

Both streams therefore have a lot of value for different use cases. If
anything you could argue that they should expose the same information
/precisely/ because they are used for different use cases. For the
logging/reading use case why would I get less information?

Perhaps controlling the information you get on the log stream could be a
command line option that allows people to have that information if they
need it, suppressing otherwise.

In our use case it is significantly less complicated to parse stdout on
the fly to see what is happening. We need to do that anyway for the
messages we output. Having to then parse stderr (say) for additional
information and merge is more complex. Redirecting the report stream to
stdout could help but then I lose the ability to get the test report
which I might want separately so send back to a build server.

I'll summarise by saying I don't think you compromise the separation
between the log and report streams by adding the missing detail to the
log stream. If anything you emphasize that the two streams are for
different purposes. They both carry the same test information but for
different means.

Jamie

>
> Gennadiy
>
>
>
>
> _______________________________________________
> Unsubscribe& other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
>


Boost list run by bdawes at acm.org, gregod at cs.rpi.edu, cpdaniel at pacbell.net, john at johnmaddock.co.uk