From: David Abrahams (dave_at_[hidden])
Date: 2007-08-04 17:42:59
This is part 1 of my analysis of our prospects for improving the Boost
release process by improving the tools. I'll post something based on
this to the wiki after y'all give me an earful about what I got wrong
I'm only considering BBv2 and cmake here, because we currently use the
former and Doug and Troy have been putting lots of effort into the
latter, and Kitware has volunteered to support it for us.
Cmake has lots of advantages over BBv2 for developers (e.g. instant
build startup time, use favorite development environment) and
packagers (e.g. one-line installer target declaration), but these
factors are more important for day-to-day development than they are
for testing. Instant build startup time would improve test turnaround
time, of course, but build startup does not dominate our testing time
yet. BBv2 also has overall speed/memory-consumption problems. These
might be addressable using the Boehm GC, but the last attempt resulted
in crashes (more investigation needed).
BBv2 has a few advantages over Cmake for testing.
1. BBv2 can build with multiple toolsets and/or variants in one
invocation. That, however, is trivially overcome by using a
wrapper script over cmake.
2. BBv2 serializes the output of simultaneous tests, which makes it
possible to run tests in parallel with -jN. This is really
important for test turnaround time. Even uniprocessors benefit
from a low N (like 2) IME, due to disk/cpu tradeoffs. The IBM
testers are currently using -j16 to great effect on their big iron.
However, output serialization is only important because both
Boost's process_jam_log and ctest/dart currently use a
parse-the-build-log strategy to assemble the test results. This
approach is hopelessly fragile in my opinion. Instead, each build
step in a test should generate an additional target that contains
the fragment of XML pertaining to that test, and the complete
result should be an XML file that is the catenation of all the
fragments. Rene is working on that feature for BBv2. I don't
expect it would take more than a couple of days to add this feature
to either build system, so the advantage of serialization may
be easily neutralized.
3. Platform/compiler-independent build specification. This one seems
pretty important on the face of it. If library authors' tests
won't run on platforms to which they don't have direct access,
we'll need to find people to port and maintain test specifications
for the other platforms.
On the other hand, very few tests are doing anything fancy when it
comes to build configuration. To understand the real impact of
this BBv2 feature, someone would have to objectively analyze the
cmake build specifications Doug and Troy have been working on for
complexity and portability. If either of those guys can be
objective, it would be fastest if they'd do the analysis.
4. Cmake's built-in programming language is even more horrible, IMO,
than bjam's. I bet you get used to it, like anything else, but it
does present a barrier to entry. On the other hand, it's my
impression that most programming jobs can be done more simply and
directly under Cmake because it lacks BBv2's
multiple-toolset/variant builds, and so has no need for virtual
Neither systems can trace #include dependencies through macros
(e.g. #include SOME_MACRO(xyz)), which occasionally leads to
inaccurate results from incremental testing. This is an important
problem that should be fixed.
Based on the above points, I'd say either build system has the
potential to work for Boost's testing needs and neither one is
substantially closer to being ideal for us (for testing). Since our
systems use BBv2 now, it makes sense to concentrate effort there.
However, if cmake were to suddenly acquire the XML fragment generation
and the analysis of point 3 above showed that portable build
specification is easily accomplished in cmake or not very important to
our testing needs, I'd probably be vote the other way.
-- Dave Abrahams Boost Consulting http://www.boost-consulting.com The Astoria Seminar ==> http://www.astoriaseminar.com
Boost list run by bdawes at acm.org, gregod at cs.rpi.edu, cpdaniel at pacbell.net, john at johnmaddock.co.uk