Re: [boost] [test] Patches to improve test log detail for better thirdparty integration

Subject: Re: [boost] [test] Patches to improve test log detail for better thirdparty integration
From: Jamie Allsop (ja11sop_at_[hidden])
Date: 2012-09-26 08:17:14

On 25/09/12 18:14, Gennadiy Rozental wrote:
> Jamie Allsop<ja11sop<at>  yahoo.co.uk>  writes:
>
>>
>> These are patches we've been using for some time locally, the recent
>> discussions highlighted the importance of pushing patches like these to
>> a wider audience.
>>
>> In summary the key changes are:
>>
>>     * https://svn.boost.org/trac/boost/ticket/7410
>>     * https://svn.boost.org/trac/boost/ticket/7397
>
> These are fine. I'll try to check it in on a weekend

Excellent thanks!

>
>>     * https://svn.boost.org/trac/boost/ticket/7417
>
> This is not a good direction IMO. This breaks clear separation between test log
> and report streams. We also do not want duplication of information in two
> streams. We should consider an approach where report stream is being produced on
> a fly (in parallel) during test execution. This might be a bit tricky and is not
> always desirable, but can be enables by special command line switch.

Interesting idea - having the report be produced on the fly might work 
well, especially if using human readable format. Essentially you are 
only a couple of regexes away from useful output. I suppose my main need 
is to be able to see test progress as:

test XXX started...
<test now progressing - could take some time>
test XXX completed - status - elapsed - assertions passed out of total

Having said all that I do not see the harm is adding this to the test 
log. Here's why. The test log already specifies when a test is started 
and when a test completes. When a test completes it already gives you 
the elapsed time. Often you *want* to see the test log output with other 
messages and so on that you write to the file. Moreover you tend to 
parse or interpret the test log on the file, tail it or whatever. 
Therefore simply adding the test status and assertion counts in addition 
to the time elapsed seems like a good fit to me.

I see the role of the report stream as being very different. In the 
report stream I see the desire to create an arbitrary report in an 
arbitrary format (like XML). The information contained in the report 
stream or the log stream is not really the issue.

In the log stream you get the logged messages *and* the basic test 
detail as a simple string so as a human you can read the log and know 
that test XXX started 'here' output these message and then completed 
after a period of time. Knowing also whether test XXX passed or failed 
after that time period and how many assertions were successfully 
executed is only a slight addition to the current detail. The key point 
is the log stream is often there to be 'read'.

The report stream contains essentially the same test information as the 
log stream then, except that it *only* contains the test status detail. 
In essence the report is more like a summary (no log messages and so on 
- just the test detail). It's primary function is to create a report 
that can then be used for later interpretation (often by a CI system) - 
not really for 'reading'.

Both streams therefore have a lot of value for different use cases. If 
anything you could argue that they should expose the same information 
/precisely/ because they are used for different use cases. For the 
logging/reading use case why would I get less information?

Perhaps controlling the information you get on the log stream could be a 
command line option that allows people to have that information if they 
need it, suppressing otherwise.

In our use case it is significantly less complicated to parse stdout on 
the fly to see what is happening. We need to do that anyway for the 
messages we output. Having to then parse stderr (say) for additional 
information and merge is more complex. Redirecting the report stream to 
stdout could help but then I lose the ability to get the test report 
which I might want separately so send back to a build server.

I'll summarise by saying I don't think you compromise the separation 
between the log and report streams by adding the missing detail to the 
log stream. If anything you emphasize that the two streams are for 
different purposes. They both carry the same test information but for 
different means.

Jamie

>
> Gennadiy
>
>
>
>
> _______________________________________________
> Unsubscribe&  other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
>