Re: [boost] Release Tools Analysis Part I: Build systems

From: Beman Dawes (bdawes_at_[hidden])
Date: 2007-08-07 10:51:53

David Abrahams wrote:

> 2. BBv2 serializes the output of simultaneous tests, which makes it
>    possible to run tests in parallel with -jN.  This is really
>    important for test turnaround time.  Even uniprocessors benefit
>    from a low N (like 2) IME, due to disk/cpu tradeoffs.  The IBM
>    testers are currently using -j16 to great effect on their big iron.
> 
>    However, output serialization is only important because both
>    Boost's process_jam_log and ctest/dart currently use a
>    parse-the-build-log strategy to assemble the test results.  This
>    approach is hopelessly fragile in my opinion.  Instead, each build
>    step in a test should generate an additional target that contains
>    the fragment of XML pertaining to that test, and the complete
>    result should be an XML file that is the catenation of all the
>    fragments.  Rene is working on that feature for BBv2.  I don't
>    expect it would take more than a couple of days to add this feature
>    to either build system, so the advantage of serialization may
>    be easily neutralized.

Agreed. Parse-the-build-log is a horrid stop-gap. "hopelessly fragile" 
doesn't even begin to describe how easily it breaks.

It was supposed to hold-the-line for a few months until BBv2 was ready. 
That was in October, 2002.

The other point of parse-the-build-log was to refine what was needed in 
the XML files. That part has been pretty successful; we now know what 
the XML needs to convey.

One element that is still missing from the XML is timings for each of 
the steps, but particularly the compile and run steps. I remain 
convinced that both wall-clock and CPU-usage timings are an important 
element of high-quality testing and test management.

> 3. Platform/compiler-independent build specification.  This one seems
>    pretty important on the face of it.  If library authors' tests
>    won't run on platforms to which they don't have direct access,
>    we'll need to find people to port and maintain test specifications
>    for the other platforms.

It isn't just that tests have to run on platforms the authors don't have 
direct access to, but also that the authors have no knowledge or desire 
to learn about build specifications on those platforms.

>    ...

> Based on the above points, I'd say either build system has the
> potential to work for Boost's testing needs and neither one is
> substantially closer to being ideal for us (for testing).  Since our
> systems use BBv2 now, it makes sense to concentrate effort there.
> However, if cmake were to suddenly acquire the XML fragment generation
> and the analysis of point 3 above showed that portable build
> specification is easily accomplished in cmake or not very important to
> our testing needs, I'd probably be vote the other way.

I'd say it isn't worth moving unless the new system has *very* 
significant advantages over what we have now.

Robustness is really important to me, and it is going to be hard to 
convince me that any system that uses parse-the-build-log is robust.

I see "portable build specification" and everything else that is a part 
of hiding the details of the underlying platform specific tools as being 
  of critical importance.

Thanks for tackling this analysis!

--Beman