|
|
Boost Interest : |
From: Doug Gregor (doug.gregor_at_[hidden])
Date: 2008-06-01 12:20:54
Hi Troy,
On Sat, May 31, 2008 at 3:50 PM, troy d. straszheim <troy_at_[hidden]> wrote:
> When run, CTest reads these lists of tests in, runs them, and
> redirects the output to logfiles. It then must scrape results of
> builds/tests out of logfiles, tries (with varying degrees of success)
> to identify errors, and posts them in large chunks.
> This log-scraping is an architectural weak point that is not going to
> go away. For instance, here is a snippet from ctests's source:
>
> static const char* cmCTestErrorMatches[] = {
> "^[Bb]us [Ee]rror",
[snip]
> [wince]
>
> As a consequence of this architecture, there are a number of things
> that we cannot easily do on any build reporting system that ingests
> only the information currently reported by ctest:
>
> === Catalogue o' Worries ===
>
> 1. Make a nifty "M of N steps completed" graph on in-progress builds
> with good resolution.
We have some of this, in the form of the completion-percentage that
CMake emits whenever it compiles something, although we could argue
that the resolution isn't very good (and, for Boost, it's at the wrong
level of granularity).
> 2. Pinpoint the source of certain errors. See
>
> http://www.cmake.org/pipermail/cmake/2008-May/022034.html
>
> for a discussion of a case where link errors are not reported. It
> is neither possible to immediately tell what went wrong, nor to
> tell where in the dependency tree one was when they occurred (one
> must infer it from the list of other targets that failed, some of
> which might also be 'unknown')
Agreed.
> 3. Finely control the rate at which build/test results are submitted. CTest
> directly supports only reporting after Configure, Build, and Test. The
> most often you can get ctest to report things is like this:
>
> ctest -D ExperimentalStart # starts a new 'tag' for a build
> ctest -D ExperimentalConfigure # runs cmake
> ctest -D ExperimentalSubmit # post results of that configure
> ctest -D ExperimentalBuild # run build
> ctest -D ExperimentalSubmit # post results of that build
> ctest -D ExperimentalTest # run tests
> ctest -D ExperimentalSubmit # post results of that test run
>
> By contrast, the natural 'post rate' for boost (also for IceCube,
> presumably also for KDE) is per-library, something like:
>
> * configure
> * post (include upcoming list of libs-to-build and libs-to-test)
> * build lib1
> * post
> * build lib2
> * post
> ...
> * build libn
> * post
> * test lib1
> * post
> * test lib2
> * post
Of course, most test-reporting systems don't deal with this level of
granularity. Bitten is different/new in this regard. (That said, it's
still a good idea!).
> 4. Tell how many *successful* build steps were executed. ctest
> reports only failures. For instance, if I run an incremental
> build and look at the results on dart, I don't know how many
> files were actually rebuilt. I often want to know this information,
> though: for instance, if the patches I committed haven't fixed
> certain test failures, I really want to be able to check that the
> tests themselves were actually rebuilt.
Interesting. What's more important to know---which files were rebuilt,
or which Subversion revision
> 5. See the actual commands executed to run certain builds. One often
> wants to do this when chasing build misconfigurations: what were
> the flags this lib was built with?
Yes, this is really, really important information whenever something
fails, and CTest/Dart/CDash don't do a good job of handling this.
> Integration-wise, testing targets aren't really first class citizens
> of the cmake-generated makefiles.
> 'make test' executes ctest in a fragile cascade of subshells.
Well, I don't know if it's fragile or not (it's never caused problems
for me), but there are other issues with not having testing targets be
first-class citizens. It's hard to run tests for just one library
without building the tests for *all* Boost libraries (as part of the
Build stage) and then running CTest directly, e.g., with "ctest -R
MPI". That's why we have the TEST_BOOST_* configuration options... but
it's still messy.
> Also, ctest hardcodes a set of 'testing models': Nightly, Weekly,
> Experimental, which are aribtrary, distracting, and couple the system
> that runs the builds (the ctest side) to the system that displays them
> (the dart/cdart/etc side).
Right, these testing models are meant to support Dart's model, which
isn't necessarily the right model for Boost. I don't think their
existence is a problem; having features that are important to one user
group but not another is the price we pay for using someone else's
software, but the benefits generally outweigh the costs.
[snip log-scraping issues, solution]
> On integration into cmake:
>
> - CMake already does something similar: Think
> CMAKE_VERBOSE_MAKEFILES and the toggleable fancy colorization
> and percent-completed display.
Right. Note that this stuff does not work in Visual Studio, which may
be an issue. I don't understand the Visual Studio model well enough to
have a good sense of whether something similar is possible. I guess in
the worst case, we have some limitations in Visual Studio or ask
regression testers to use NMAKE
> - CMake already builds against its own curl or a system curl. (It is
> used by ctest). So the code one needs to post results is there
> already, if you wanted to code up this wrapper in C++ and push it upstream
> to the cmake project. I can see starting with some small python scripts.
CTest has full XML-RPC support on top of curl, so all of the
functionality is there in CMake... we might need to tweak things to
expose that functionality more directly.
> - The 'wrapper' could conceivably be coded up in C++ and
> built/installed by the cmake distribution. Maybe distributing
> a python script is just as easy.
Sure. At this point, we're not requiring Python in the CMake setup,
but it's okay if we need to require Python.
> I think that covers the business of getting at the build results in a non-lossy
> fashion.
>
> Referring to Catalogue O' Worries entry #3, ideally one would like to
> post results
>
> - at the end of the build of each component (e.g. libBAR, libFOO),
> - at the end of the build of the *tests* for this component,
> - and at the end of the *run* of the tests for this component
>
> and to do so recursively, i.e. if libBAR depends on libFOO, the
> build/test/post of libFOO will be done automagically.
Just the build+post of libFOO will be done, not the testing of libFOO, right?
> This implies
> peppering the build dependency tree with intermediate targets
> that collect results and post them.
Okay.
CMake has most of the facilities we need to move testing of Boost into
the build, which is one of the steps in this process. The
Boost-specific test macros (boost_test_run, boost_test_compile, etc.)
could be changed to use add_custom_command, to make testing a part of
the normal build process. I'm worried about some of the auxiliary
issues---handling timeouts, avoiding those "would you like to start a
debugger?" dialogs on Windows, dealing with run-fail and compile-fail
tests, etc.---that CTest already handles. I guess some or all of that
could be dealt with in the wrapper that's used to generate XML for the
build output.
- Doug