From: Douglas Gregor (gregod_at_[hidden])
Date: 2002-01-21 11:36:18
The use of a Dart dashboard would meet or exceed Boost's regression testing
needs. Brad King announced an initial regression-testing system based on Jam
and Dart, and Kitware has agreed to host the Boost dashboard here:
To see the results of the Boost regression testing done by Dart, please see:
You will see that nightly testing is being done on a few platforms and
compilers - error logs, summaries, etc. are available for all platforms. For
a glimpse of what the Dart-based system could become, look at one of the
other Dart dashboards, such as the VTK dashboard at:
I'll reply point-by-point below, and try to describe how Dart fulfills your
> On of the most glaring weaknesses in our current regression test approach
> is that if a test is failing on a compiler/platform not used by the
> developer, it may be a long time before the test failure comes to the
> developer's attention.
Dart is based (primarily) on nightly builds; each Dart client will perform a
CVS update at 2:00am and then perform full regression testing on that update.
So the lag in determining the results of a change committed to CVS is at
worst about 24 hours.
> A second weakness is that the current test reporting by platform is hard to
> use for those who want to see the results on all all platforms.
Dart displays all nightly build summaries for all platforms/compilers/etc on
one web page, so it is very easy to get a glimpse of what is happening.
> One possibility would be to write a program which merges the regression
> tests tables for all platforms into a single giant table, and then make
> that table available daily (and on the web site for each release).
This is essentially what Dart does, though the summary web page is
regenerated hourly to catch any new incoming regression test results.
> Would this be a good way to address the problem?
It's worked well for others before. Again, see the VTK dashboard.
Overall, the implementation of the Dart/Jam regression testing is quite new
and incomplete. However, Brad has been working on a much refined version that
can coexist nicely with the current regression-testing system, and I've been
squashing bugs for the nightly builds I run whenever I find the time.
Brad at one point mentioned an interesting side effect of regression testing
with Dart: performing releases becomes almost trivial, because one need only
pick a day where the dashboard is completely green (i.e., no errors, although
you surely won't see that on the Boost dashboard at the moment).
Boost list run by bdawes at acm.org, gregod at cs.rpi.edu, cpdaniel at pacbell.net, john at johnmaddock.co.uk