Re: [boost] Release Tools Analysis Part II: Developer Feedback Systems

From: John Maddock (john_at_[hidden])
Date: 2007-08-08 12:15:09

David Abrahams wrote:
> Boost's Systems
> ---------------
>
> The major problems with our current feedback systems, AFAICT, are
> fragility and poor user interface.

100% agreement, it's not fault of metacomm: the current structure just can't 
cope with the volume of data generated these days.

> Recommendations
> ---------------
>
> Our web-based regression display system needs to be redesigned and
> rewritten.  It was evolved from a state where we had far fewer
> libraries, platforms, and testers, and is burdened with UI ideas that
> only work in that smaller context.  I suggest we start with as minimal
> a display as we think we can get away with: the front status reporting
> page should be both useful and easily-grasped.
>
> IMO the logical approach is to do this rewrite as a Trac plugin,
> because of the obvious opportunities to integrate test reports with
> other Trac functions (e.g. linking error messages to the source
> browser, changeset views, etc.), because the Trac database can be used
> to maintain the kind of history of test results that Dart manages, and
> because Trac contains a nice builtin mechanism for
> generating/displaying reports of all kinds.  In my conversations with
> the Kitware guys, when we've discussed how Dart could accomodate
> Boost's needs, I've repeatedly pushed them in the direction of
> rebuilding Dart as a Trac plugin, but I don't think they "get it" yet.
>
> I have some experience writing Trac plugins and would be willing to
> contribute expertise and labor in this area.  However, I know that
> we also need some serious web-UI design, and many other people are
> much more skilled in that area than I am.  I don't want to waste my
> own time doing badly what others could do well and more quickly, so
> I'll need help.

Just thinking out loud here, but I've always thought that our test results 
should be collected in a database: something like each test gets an XML 
result file describing the test result, which then gets logged in the 
database.  The display application would query the database, maybe in 
real-time for specific queries and present the results etc.  Thinking 
somewhat outside the box here ... but could SVN be conscripted for this 
purpose, yes, OK I know it's an abuse of SVN, but basically our needs are 
quite simple:

* Log the output from each build step and store it somewhere, with 
incremental builds much of this information would rairly change. In fact 
even if the test is rebuilt/rerun the chances are that logged data won't 
actually change.
* Log the status of each test: pass or fail.
* Log the date and time of the last test.

So what would happen if build logs for each test were stored in an SVN tree 
set aside for the purpose, with pass/fail and date/time status stored as SVN 
properties?  Could this be automated, from within bjam or CMake or whatever?

Of course we're in serious danger of getting into the tool writing business 
again here ....

John.