[boost] proposal - modularize Boost build system

Subject: [boost] proposal - modularize Boost build system
From: Stefan Seefeld (stefan_at_[hidden])
Date: 2017-06-18 16:09:59

One of Boost's bottlenecks is its monolithic build infrastructure. It
would be beneficial for many different use cases (testing, packaging,
etc.) to decouple the building of Boost libraries (well, those that
require building, see the listing in
http://www.boost.org/doc/libs/1_64_0/more/getting_started/unix-variants.html#header-only-libraries)
such that they may optionally be built stand-alone.

At present, building a Boost library requires the entire
(super-)repository to be checked out, and the build logic itself
involves traversing the entire source tree, looking for a "Jamroot".

What would be beneficial to many would be a workflow like this:

1) Have a development environment with (some version of) Boost
pre-installed (or at least the parts that the library being built
depends on).

2) Check out a single Boost repository (e.g.,
https://github.com/boostorg/python)

3) Invoke a command to build it (if there is anything to build)

4) Invoke a command to test it

5) Invoke a command to install it

6) Invoke a command to package it (optional)

While it's of course already possible to do all the above by adding
support for another build infrastructure (point in case: Boost.Python
right now uses SCons for this very reason), this means duplication of
effort, as Boost as a whole is still built, tested (and even packaged
for binary packages) using Boost.Build, meaning I need to maintain two
sets of build infrastructure.

This proposal thus has two parts:

1) Make the nested build logic independent of the outer bits, so
individual libraries can be built stand-alone (for example, using b2 by
adding a `--with-boost` set of options to point to the location of the
prerequisite parts of Boost).

2) Define a clear interface the outer build logic will use to invoke the
nested build commands.

Note that for my own case above (Boost.Python), providing 2) would be
enough, i.e. by having a way for me to "plug in" my SCons commands to
build and test Boost.Python could obsolete the existing Boost.Build
logic as defined in Boost.Python (as well as the prerequisite parts in
Boost.Build's "python" module).

However, as my proposal is *not* actually advocating to move away from
Boost.Build, but rather to modularize it, I think 1) is essential to let
everyone else (who may not be inclined to use anything other than
Boost.Build) to also take advantage of modularization.

While I'd rather avoid delving into the technical details of how this
could possibly be implemented (if you really must, *please* do so in a
new thread !), let me outline a few use-cases that would be made
possible by the above:

* individual projects would be free to switch to their preferred
infrastructure, including CMake, SCons, etc., if they so wish

* individual projects could be much more easily developed and contributed to

* individual projects could be much more easily tested, notably in CI
environments

* individual projects could be much more easily packaged

All of the above advantages are *huge*, and reflect real-world needs,
and the technical issues to solve these are all minor. The question
really is whether there is enough will to move into that direction.

I'd be very happy to participate in the work needed to implement this.
But first we need to agree that this is where we want to go.

Thanks,
        Stefan

-- 
      ...ich hab' noch einen Koffer in Berlin...