Re: [Boost-cmake] Regression Test Reporting

Subject: Re: [Boost-cmake] Regression Test Reporting
From: troy d. straszheim (troy_at_[hidden])
Date: 2008-12-02 10:37:23

Hey Bill,

Bill Hoffman wrote:
>
> That said, we have a large software project with similar needs as Boost 
> interested in adding some features that might change that.  Of course 
> none of this is implemented yet, but in the next 3 to 4 months CDash 
> maybe in a much better position to work with Boost.   I will keep you 
> posted.
> 
> If you had some specifics, it might be helpful to guide us as we add new 
> features to CDash.  So, if either you and/or Troy could send me a 
> summary of features that you would like to see, it would be helpful.  I 
> remember some of your requests from when we talked in Boston, but that 
> was a while ago...

Discussion is here:

   http://thread.gmane.org/gmane.comp.lib.boost.cmake/4/focus=5

and here:

   http://thread.gmane.org/gmane.comp.lib.boost.cmake/10

One thing that has changed since that discussion is
tests-as-first-class-targets:  for boost, at least, the extra cost
in generation time and time-to-build is prohibitive, and the
regex-selection style of ctest (-R mylibrary) is more straightforward
and functional.  (You get situations where you want to specify that certain
tests are run in order, and doing this with make-target-dependencies is a big
hassle).

Note that in the ctest/cdash model, ctest scrapes logfiles and watches for
timeouts, in addition to selecting and running tests (e.g. via regex);
in the 'other' model, a python driver script separate from the 'ctest' executable
captures output, so the 'ctest' executable is really simple
(you can code one in python in 15 minutes).

> The features we are thinking about adding include:
> 
> - extend ctest to be able to build/report one package at a time

This implies closer integration of ctest with first-class
make targets, no?

> - extend cdash to support the display of subsets of packages and 
> dashboards.

IMV the front end stuff is not nearly as important as a robust capture
of all output of the build/test run.   That may just be me.

> Basically, handle an over all view (like now), and a sub-project view of 
> test results.  So, if you are the developer of package A, you do not 
> need to be bothered with failures in package B if you do not directly 
> depend on it.  As I recall that was the big thing CDash was lacking from 
> the Boost perspective.

Again I think it is deeper than this, but let's see what you make of those
mailing list threads...

-t