Re: [boost] Release Tools Analysis Part I: Build systems

From: David Abrahams (dave_at_[hidden])
Date: 2007-08-05 13:51:00

on Sun Aug 05 2007, "Timothy M. Shead" <tshead-AT-k-3d.com> wrote:

> David Abrahams wrote:
>
>>    However, output serialization is only important because both
>>    Boost's process_jam_log and ctest/dart currently use a
>>    parse-the-build-log strategy to assemble the test results.  This
>>    approach is hopelessly fragile in my opinion.  Instead, each build
>>    step in a test should generate an additional target that contains
>>    the fragment of XML pertaining to that test, and the complete
>>    result should be an XML file that is the catenation of all the
>>    fragments.  Rene is working on that feature for BBv2.  I don't
>>    expect it would take more than a couple of days to add this feature
>>    to either build system, so the advantage of serialization may
>>    be easily neutralized.
>
> There may be some semantics here that I'm missing, but I think ctest
> is already doing exactly what you describe:

Bill Hoffman (of Kitware) himself told me that parallel builds don't
work with ctest because of output interleaving issues.  Was he
mistaken?

> * ctest checks the process return value from each test.  A nonzero
> value equals test failure (unless the test is marked as an expected
> failure).
>
> * ctest (optionally) parses the test's output (stdout/stderr) using a
> configurable regular expression.  If there is a match, the test
> fails. The intent is to catch errors reported by some standard logging
> mechanism, if such a mechanism exists, e.g. messages of the form
> "CRITICAL ERROR: blah blah blah ...".  You can disable this feature if
> you don't want it.

All of the above is irrelevant to the question at hand, I think.

> * ctest produces an XML file that describes the results of each test,
> including pass/fail, execution time, and test output (stdout/stderr).

A separate XML file for each test?  Can it generate these XML files if
the tests are run in parallel?  That's the key (relevant) issue.

-- 
Dave Abrahams
Boost Consulting
http://www.boost-consulting.com
The Astoria Seminar ==> http://www.astoriaseminar.com