Re: [boost] Best way to notify of regresssion test failures?

From: Douglas Gregor (gregod_at_[hidden])
Date: 2002-01-21 16:26:29

On Monday 21 January 2002 03:40 pm, you wrote:
> At 11:36 AM 1/21/2002, Douglas Gregor wrote:
>  >  The use of a Dart dashboard would meet or exceed Boost's regression
>  >testing needs.
>
> I'm interested in Dart dashboard, but don't see how Boost could give up
> control of our regression testing to a third-party.

I'm not quite sure what you mean by "control" with respect to regression 
testing. The Dart server is run at Kitware, but the server does not perform 
any regression testing itself; it merely collects regression testing data 
submitted by the Dart clients. If by "control" you mean the ability to 
maintain the server, I think that is largely irrelevant in this case. Kitware 
runs Dart dashboards for several of their own projects on the same server 
that the Boost dashboard will be available. They can't afford to have their 
dashboards down, so problems will be fixed as fast as possible. In the worst 
case, Dart is open source software and the Boost dashboard could be moved 
elsewhere.

The actual regression testing is done by the Dart client. Anyone can setup a 
Dart client and submit regression tests. At the moment, Kitware runs one 
nightly regression test, and I run three others on different platforms. Dart 
clients will perform a "cvs update" every night, so all nightly build entries 
posted on the dashboard for a specific day will be for the same version of 
the Boost source (this is a _huge_ problem for me now, with the current 
regression testing system; I look at the tests, exclaim "I broke WHAT?", and 
then realize that the tests are two months old).

I should just ask: what do you mean by "control"? I see no problem with the 
server side of a Dart-based system, and Dart clients are in the hands of 
those that have the hardware to run regression tests on.

> While it may not be obvious to all Boost members, what tends to torment
> Boost moderators (or at least me) are the functions where someone else has
> control of our resources.  YahooGroups.  SourceForge.

I understand this completely, but I think regression testing is a slightly 
different problem. It's _impossible_ to control all of the resources for 
regression testing unless Boost had an infinite number of platform/compiler 
combinations in someone's basement. 

> Toolchains that aren't covered by the Boost distribution are also a
> problem.  I'm setting up a new machine right now.  To run various Boost
> procedures, I need to install five C++ compilers, Cygwin, Python, WinCvs,
> and be able to cope with Windows/bat, Python, and bash scripts.  People
> would like change logs. The only scripts I can find are in Perl.  So add
> that to the list.  I'm not at all happy to even think about anything else.

This was brought up earlier: Dart requires Tcl, so performing all regression 
testing with Dart would require everyone to install both Dart and Tcl 
locally. The solution is to split regression testing into a front-end and a 
back-end: the front-end is just a common method of specifying regression 
tests in Jam, and the back-end does the actual work. Two backends would be 
supported:
  1) A Dart client backend that is used for anything submitted to the Dart 
dashboard.
  2) A "local testing" backend that works like status/regression.cpp. This 
would generate the HTML files to which we are accustomed, and would not 
require Tcl or Dart.

Brad's posted some prototypes for the front-end, and I believe he's also 
started converting the Jamfiles we created into a Dart-based back-end.

The result: only those running tests to be submitted to the Dashboard need to 
have Tcl and Dart. Nobody needs to have five C++ compilers, because each 
build variant can be tested and submitted separately. 

        Doug