[boost] Release Tools Analysis Part I: Build systems

From: David Abrahams (dave_at_[hidden])
Date: 2007-08-04 17:42:59

This is part 1 of my analysis of our prospects for improving the Boost
release process by improving the tools.  I'll post something based on
this to the wiki after y'all give me an earful about what I got wrong
or overlooked.

Build System
============

I'm only considering BBv2 and cmake here, because we currently use the
former and Doug and Troy have been putting lots of effort into the
latter, and Kitware has volunteered to support it for us.

Cmake has lots of advantages over BBv2 for developers (e.g. instant
build startup time, use favorite development environment) and
packagers (e.g. one-line installer target declaration), but these
factors are more important for day-to-day development than they are
for testing.  Instant build startup time would improve test turnaround
time, of course, but build startup does not dominate our testing time
yet.  BBv2 also has overall speed/memory-consumption problems.  These
might be addressable using the Boehm GC, but the last attempt resulted
in crashes (more investigation needed).

BBv2 has a few advantages over Cmake for testing.  

1. BBv2 can build with multiple toolsets and/or variants in one
   invocation.  That, however, is trivially overcome by using a
   wrapper script over cmake.

2. BBv2 serializes the output of simultaneous tests, which makes it
   possible to run tests in parallel with -jN.  This is really
   important for test turnaround time.  Even uniprocessors benefit
   from a low N (like 2) IME, due to disk/cpu tradeoffs.  The IBM
   testers are currently using -j16 to great effect on their big iron.

   However, output serialization is only important because both
   Boost's process_jam_log and ctest/dart currently use a
   parse-the-build-log strategy to assemble the test results.  This
   approach is hopelessly fragile in my opinion.  Instead, each build
   step in a test should generate an additional target that contains
   the fragment of XML pertaining to that test, and the complete
   result should be an XML file that is the catenation of all the
   fragments.  Rene is working on that feature for BBv2.  I don't
   expect it would take more than a couple of days to add this feature
   to either build system, so the advantage of serialization may
   be easily neutralized.

3. Platform/compiler-independent build specification.  This one seems
   pretty important on the face of it.  If library authors' tests
   won't run on platforms to which they don't have direct access,
   we'll need to find people to port and maintain test specifications
   for the other platforms.

   On the other hand, very few tests are doing anything fancy when it
   comes to build configuration.  To understand the real impact of
   this BBv2 feature, someone would have to objectively analyze the
   cmake build specifications Doug and Troy have been working on for
   complexity and portability.  If either of those guys can be
   objective, it would be fastest if they'd do the analysis.

4. Cmake's built-in programming language is even more horrible, IMO,
   than bjam's.  I bet you get used to it, like anything else, but it
   does present a barrier to entry.  On the other hand, it's my
   impression that most programming jobs can be done more simply and
   directly under Cmake because it lacks BBv2's
   multiple-toolset/variant builds, and so has no need for virtual
   targets.

Neither systems can trace #include dependencies through macros
(e.g. #include SOME_MACRO(xyz)), which occasionally leads to
inaccurate results from incremental testing.  This is an important
problem that should be fixed.

Based on the above points, I'd say either build system has the
potential to work for Boost's testing needs and neither one is
substantially closer to being ideal for us (for testing).  Since our
systems use BBv2 now, it makes sense to concentrate effort there.
However, if cmake were to suddenly acquire the XML fragment generation
and the analysis of point 3 above showed that portable build
specification is easily accomplished in cmake or not very important to
our testing needs, I'd probably be vote the other way.

-- 
Dave Abrahams
Boost Consulting
http://www.boost-consulting.com
The Astoria Seminar ==> http://www.astoriaseminar.com