[boost] Re: Regression report for Releases?

From: Aleksey Gurtovoy (agurtovoy_at_[hidden])
Date: 2005-03-21 08:17:18

Jeff Garland writes:
>>   - Regressions from the previous release are nice to know but less
>>     important.  I realize we show both in one report, but this may
>>     help us adjust our emphasis or coloring (maybe it's already
>>     perfect in the user report; I don't know)
>
> In fact, I think from the user perspective the question really goes something
> like: "I'm using Intel 8.1 on Windows and Linux and I want to use Python,
> smart_ptr, and serialization -- can I expect these libraries to work with
> release xyz?"  And several variations on that theme. So in an "ideal world"
> scenario I would have a web form where the user could enter her needs and a
> script would filter the regression results down to the set of interest for the
> user. 

Same for a developer, actually, who usually wants to track only a
couple of libraries. The problem with such scheme is that it by
definition bumps up requirements on the reports' hosting site. Right
now the pages are nothing but plain HTML. Ideally, to handle the kind
of dynamic requests you describe in real time we need a database
backend, and that severely limits where the reports can be hosted. If
we as a group decide that the benefits of having something like this
outweigh the downside, this can be pulled off quite easily.

>
>>   - A health report for the current state of the repository should
>>     always be available on the website.
>> 
>>   - Regressions from the previous release are crucial to know also
>> 
>>   - When we branch for a release, we absolutely must track the 
>>     release branch, but we also should be continuing to display 
>>     the health of the trunk
>
> Agree -- I think the big blocker here is expanding the set of regression
> testers during this period.  Another factor is that the regression
> compilers/platforms tested between releases is not really stable.  It has been
> growing, so we have now have 'new results' that can't really 'diff' from the
> last release.  For example, we now have a regression tester for Solaris which
> we didn't have in 1.32.  I'm not sure that's obvious from the way we display
> results.

Suggestions are welcome.

-- 
Aleksey Gurtovoy
MetaCommunications Engineering