Subject: Re: [boost] proposal - modularize Boost build system
From: Raffi Enficiaud (raffi.enficiaud_at_[hidden])
Date: 2017-06-18 18:48:56
Le 18.06.17 à 18:09, Stefan Seefeld via Boost a écrit :
> One of Boost's bottlenecks is its monolithic build infrastructure. It
> would be beneficial for many different use cases (testing, packaging,
> etc.) to decouple the building of Boost libraries (well, those that
> require building, see the listing in
> such that they may optionally be built stand-alone.
> At present, building a Boost library requires the entire
> (super-)repository to be checked out, and the build logic itself
> involves traversing the entire source tree, looking for a "Jamroot".
> What would be beneficial to many would be a workflow like this:
> 1) Have a development environment with (some version of) Boost
> pre-installed (or at least the parts that the library being built
> depends on).
If you want to easily distribute boost libraries independently, having
boost preinstalled (or already cloned or already downloaded from a
release) is not exactly what I would like to see as a user.
So my user story would be:
* I have a compiler, and the prerequisite for building boost.whatever,
which are stated by the boost.whatever readme.txt file, and that should
be light enough, or at least the minimal thing for building stuff for
the platform I am running.
* I git clone
* I build, and possibly run the tests to feel confident.
From my user perspective, all the rest is "implementation details".
This is the type of build/run I can see for other popular libraries, I
wish we get there at some point with boost.
As a library maintainer, I can say that boost.test is having hard time
to compete with other unit testing libraries such as Catch or google
test, not because of the number of files in boost.test itself, because
you need to pull a full boost release to run it, which is another level
Packaging is out of the user story to my opinion, and there are nice
people very good at this. Also if installation is easy, packaging is
less important I would say (IMO).
Installation depends and can be complicated: do we want this
installation to live with another existing one (like the superproject,
or the one shipped with a eg. Debian package) ? Do we want to support
installations in user folders? etc etc When I look at all the options
that PIP INSTALL provides me with, I am sometime lost.
Now if I take my developer hat, it seems to me that there is a different
approach to take depending on the environment the library is built from:
1. I build from within boost super project
2. I build by considering boost.whatever as an entry point, and possibly
point to an existing boost installation for dependencies
3. I build on a naked system where only the build tools are installed.
What I describe above is the very simplest use case, and even with that
this is not clear to me the approach to take.
Also, I would like to point out the annoyance for users that have to
deal with the possibly many tools I have to install. You are mentioning
SCons, and I recall myself having looked at this in early 2004 and never
looked at it since. As a user, I do not want to deal with this, that is
very specific to one library. If I need to install yet-another-tool just
to satisfy some library, then sometimes this is just a no-go (sometimes
not, I am currently looking at Bazel ... but I am kind of forced to to
The CMake thread mentioned started mentioning this, and CMake is
gathering some consensus because also, it is defacto already installed
on many of the systems developer use today. So I see Cmake not only as a
tool for building, testing etc, but also as the extra tool of less
There is also something a bit awkward I wanted to indicate, but we just
forgot to mention one of the most important tools so far: the **compiler**.
We cannot achieve any modularity if there is no agreement on what should
be supported, and this definitely is the most important variable to take
into account in the dependency graph of the boost.whatever libraries. If
I consider only boost.test, a lot of dependencies can just be removed
just by considering C++11, which in turn would lower dramatically the
user overhead in order to use the library.
> 2) Check out a single Boost repository (e.g.,
> 3) Invoke a command to build it (if there is anything to build)
> 4) Invoke a command to test it
> 5) Invoke a command to install it
> 6) Invoke a command to package it (optional)
> While it's of course already possible to do all the above by adding
> support for another build infrastructure (point in case: Boost.Python
> right now uses SCons for this very reason), this means duplication of
> effort, as Boost as a whole is still built, tested (and even packaged
> for binary packages) using Boost.Build, meaning I need to maintain two
> sets of build infrastructure.
> This proposal thus has two parts:
> 1) Make the nested build logic independent of the outer bits, so
> individual libraries can be built stand-alone (for example, using b2 by
> adding a `--with-boost` set of options to point to the location of the
> prerequisite parts of Boost).
> 2) Define a clear interface the outer build logic will use to invoke the
> nested build commands.
> Note that for my own case above (Boost.Python), providing 2) would be
> enough, i.e. by having a way for me to "plug in" my SCons commands to
> build and test Boost.Python could obsolete the existing Boost.Build
> logic as defined in Boost.Python (as well as the prerequisite parts in
> Boost.Build's "python" module).
> However, as my proposal is *not* actually advocating to move away from
> Boost.Build, but rather to modularize it, I think 1) is essential to let
> everyone else (who may not be inclined to use anything other than
> Boost.Build) to also take advantage of modularization.
> While I'd rather avoid delving into the technical details of how this
> could possibly be implemented (if you really must, *please* do so in a
> new thread !), let me outline a few use-cases that would be made
> possible by the above:
> * individual projects would be free to switch to their preferred
> infrastructure, including CMake, SCons, etc., if they so wish
> * individual projects could be much more easily developed and contributed to
> * individual projects could be much more easily tested, notably in CI
> * individual projects could be much more easily packaged
> All of the above advantages are *huge*, and reflect real-world needs,
> and the technical issues to solve these are all minor. The question
> really is whether there is enough will to move into that direction.
> I'd be very happy to participate in the work needed to implement this.
> But first we need to agree that this is where we want to go.