Re: [boost] Release Tools Analysis Part II: Developer Feedback Systems

From: Jason Sankey (jason_at_[hidden])
Date: 2007-09-12 09:23:31

David Abrahams wrote:
> on Thu Sep 06 2007, Jason Sankey <jason-AT-zutubi.com> wrote:
> 
>> David Abrahams wrote:
>>> on Wed Sep 05 2007, Jason Sankey <jason-AT-zutubi.com> wrote:
>>>
>>>> Certainly, this is the idea in general.  Software development teams are 
>>>> usually capable of creating their own automated build system, but it 
>>>> costs a lot of labour that is better spent on other things.  The only 
>>>> real advantage to rolling your own is the complete customisability.  I 
>>>> am confident that a lot of what you need will come out of the box with 
>>>> Pulse.  Also, since part of the idea from our end is to push the 
>>>> boundaries of Pulse, we will be available to add features that are required.
>>>>
>>>> I can't promise immediate addition of all requested features (it is 
>>>> clearly not practical) but if there are showstoppers we will address 
>>>> them.
>>> The main things that absolutely need to be there are:
>>>
>>> 1. support for the explicit/expected failure markup that is currently
>>>    stored at http://boost.org/status/explicit-failures-markup.xml and
>>>    integrated with our results display as described in my 
>>>    posting at the beginning of this part of the thread.
>> OK, as indicated this would need to be added.  I will look into this 
>> after I have an initial server running.
> 
> Great, looking forward to this.
> 
> For what it's worth, making results reporting reliable has become a
> critical problem for us, so the faster you can do get something
> working, the better.  And, if there's going to be a problem, please let
> us know ASAP so we can start investigating other solutions right away.

OK.  I am underway, having added boost jam support and looking at the 
best way to gather and integrate the test results.  I was hoping that 
there would be a way to run a full build with one or more XML test 
reports generated (since I know that Boost.Test supports XML reporting). 
  Looking more closely I see that the current regression testing process 
uses the normal test report format, which I can also integrate if 
generating XML proves difficult.  I'm still examining the current build 
process to try and understand the best way to do it, but should be up 
and running soon.

As an aside, the output from boost jam is somewhat hostile to 
post-processing.  One feature of Pulse is the ability to pull 
interesting information like warnings and errors out of your build log 
so we can summarise and highlight them in the UI.  With tools like make 
and GCC this is fairly easy as they have a predictable and uniform error 
message output.  Playing with boost jam I notice that the error messages 
are quite diverse and hard to predict.  Although I added post-processing 
rules for the errors I found, it might be worth looking into making the 
output more machine-friendly - not just for my sake, but for any tools 
that might want to process the output.

Cheers,
Jason