Re: [boost] Best way to notify of regresssion test failures?

From: Douglas Gregor (gregod_at_[hidden])
Date: 2002-01-21 11:36:18

Beman,
  The use of a Dart dashboard would meet or exceed Boost's regression testing 
needs. Brad King announced an initial regression-testing system based on Jam 
and Dart, and Kitware has agreed to host the Boost dashboard here:
  http://groups.yahoo.com/group/boost/message/22558

  To see the results of the Boost regression testing done by Dart, please see:
http://public.kitware.com/dashboard.php?name=boost

  You will see that nightly testing is being done on a few platforms and 
compilers - error logs, summaries, etc. are available for all platforms. For 
a glimpse of what the Dart-based system could become, look at one of the 
other Dart dashboards, such as the VTK dashboard at:
  http://public.kitware.com/dashboard.php?name=vtk
  
  I'll reply point-by-point below, and try to describe how Dart fulfills your 
requirements.

> On of the most glaring weaknesses in our current regression test approach
> is that if a test is failing on a compiler/platform not used by the
> developer, it may be a long time before the test failure comes to the
> developer's attention.

Dart is based (primarily) on nightly builds; each Dart client will perform a 
CVS update at 2:00am and then perform full regression testing on that update. 
So the lag in determining the results of a change committed to CVS is at 
worst about 24 hours.

> A second weakness is that the current test reporting by platform is hard to
> use for those who want to see the results on all all platforms.

Dart displays all nightly build summaries for all platforms/compilers/etc on 
one web page, so it is very easy to get a glimpse of what is happening. 

> One possibility would be to write a program which merges the regression
> tests tables for all platforms into a single giant table, and then make
> that table available daily (and on the web site for each release).

This is essentially what Dart does, though the summary web page is 
regenerated hourly to catch any new incoming regression test results. 

> Would this be a good way to address the problem?

It's worked well for others before. Again, see the VTK dashboard.

Overall, the implementation of the Dart/Jam regression testing is quite new 
and incomplete. However, Brad has been working on a much refined version that 
can coexist nicely with the current regression-testing system, and I've been 
squashing bugs for the nightly builds I run whenever I find the time. 

Brad at one point mentioned an interesting side effect of regression testing 
with Dart: performing releases becomes almost trivial, because one need only 
pick a day where the dashboard is completely green (i.e., no errors, although 
you surely won't see that on the Boost dashboard at the moment).

        Doug