Re: [boost] proposal - modularize Boost build system

Subject: Re: [boost] proposal - modularize Boost build system
From: Raffi Enficiaud (raffi.enficiaud_at_[hidden])
Date: 2017-06-18 18:48:56

Le 18.06.17 à 18:09, Stefan Seefeld via Boost a écrit :
> One of Boost's bottlenecks is its monolithic build infrastructure. It
> would be beneficial for many different use cases (testing, packaging,
> etc.) to decouple the building of Boost libraries (well, those that
> require building, see the listing in
> http://www.boost.org/doc/libs/1_64_0/more/getting_started/unix-variants.html#header-only-libraries)
> such that they may optionally be built stand-alone.
>
> At present, building a Boost library requires the entire
> (super-)repository to be checked out, and the build logic itself
> involves traversing the entire source tree, looking for a "Jamroot".
>
> What would be beneficial to many would be a workflow like this:
>
> 1) Have a development environment with (some version of) Boost
> pre-installed (or at least the parts that the library being built
> depends on).

If you want to easily distribute boost libraries independently, having 
boost preinstalled (or already cloned or already downloaded from a 
release) is not exactly what I would like to see as a user.

So my user story would be:
* I have a compiler, and the prerequisite for building boost.whatever, 
which are stated by the boost.whatever readme.txt file, and that should 
be light enough, or at least the minimal thing for building stuff for 
the platform I am running.
* I git clone
* I build, and possibly run the tests to feel confident.

 From my user perspective, all the rest is "implementation details".
This is the type of build/run I can see for other popular libraries, I 
wish we get there at some point with boost.

As a library maintainer, I can say that boost.test is having hard time 
to compete with other unit testing libraries such as Catch or google 
test, not because of the number of files in boost.test itself, because 
you need to pull a full boost release to run it, which is another level 
of magnitude.

Packaging is out of the user story to my opinion, and there are nice 
people very good at this. Also if installation is easy, packaging is 
less important I would say (IMO).
Installation depends and can be complicated: do we want this 
installation to live with another existing one (like the superproject, 
or the one shipped with a eg. Debian package) ? Do we want to support 
installations in user folders? etc etc When I look at all the options 
that PIP INSTALL provides me with, I am sometime lost.

Now if I take my developer hat, it seems to me that there is a different 
approach to take depending on the environment the library is built from:

1. I build from within boost super project
2. I build by considering boost.whatever as an entry point, and possibly 
point to an existing boost installation for dependencies
3. I build on a naked system where only the build tools are installed.

What I describe above is the very simplest use case, and even with that 
this is not clear to me the approach to take.

Also, I would like to point out the annoyance for users that have to 
deal with the possibly many tools I have to install. You are mentioning 
SCons, and I recall myself having looked at this in early 2004 and never 
looked at it since. As a user, I do not want to deal with this, that is 
very specific to one library. If I need to install yet-another-tool just 
to satisfy some library, then sometimes this is just a no-go (sometimes 
not, I am currently looking at Bazel ... but I am kind of forced to to 
build TensorFlow).

The CMake thread mentioned started mentioning this, and CMake is 
gathering some consensus because also, it is defacto already installed 
on many of the systems developer use today. So I see Cmake not only as a 
tool for building, testing etc, but also as the extra tool of less 
annoyance.

There is also something a bit awkward I wanted to indicate, but we just 
forgot to mention one of the most important tools so far: the **compiler**.

We cannot achieve any modularity if there is no agreement on what should 
be supported, and this definitely is the most important variable to take 
into account in the dependency graph of the boost.whatever libraries. If 
I consider only boost.test, a lot of dependencies can just be removed 
just by considering C++11, which in turn would lower dramatically the 
user overhead in order to use the library.

Raffi

> 2) Check out a single Boost repository (e.g.,
> https://github.com/boostorg/python)
>
> 3) Invoke a command to build it (if there is anything to build)
>
> 4) Invoke a command to test it
>
> 5) Invoke a command to install it
>
> 6) Invoke a command to package it (optional)
>
>
> While it's of course already possible to do all the above by adding
> support for another build infrastructure (point in case: Boost.Python
> right now uses SCons for this very reason), this means duplication of
> effort, as Boost as a whole is still built, tested (and even packaged
> for binary packages) using Boost.Build, meaning I need to maintain two
> sets of build infrastructure.
>
> This proposal thus has two parts:
>
> 1) Make the nested build logic independent of the outer bits, so
> individual libraries can be built stand-alone (for example, using b2 by
> adding a `--with-boost` set of options to point to the location of the
> prerequisite parts of Boost).
>
> 2) Define a clear interface the outer build logic will use to invoke the
> nested build commands.
>
>
> Note that for my own case above (Boost.Python), providing 2) would be
> enough, i.e. by having a way for me to "plug in" my SCons commands to
> build and test Boost.Python could obsolete the existing Boost.Build
> logic as defined in Boost.Python (as well as the prerequisite parts in
> Boost.Build's "python" module).
>
> However, as my proposal is *not* actually advocating to move away from
> Boost.Build, but rather to modularize it, I think 1) is essential to let
> everyone else (who may not be inclined to use anything other than
> Boost.Build) to also take advantage of modularization.
>
>
> While I'd rather avoid delving into the technical details of how this
> could possibly be implemented (if you really must, *please* do so in a
> new thread !), let me outline a few use-cases that would be made
> possible by the above:
>
>
> * individual projects would be free to switch to their preferred
> infrastructure, including CMake, SCons, etc., if they so wish
>
> * individual projects could be much more easily developed and contributed to
>
> * individual projects could be much more easily tested, notably in CI
> environments
>
> * individual projects could be much more easily packaged
>
>
> All of the above advantages are *huge*, and reflect real-world needs,
> and the technical issues to solve these are all minor. The question
> really is whether there is enough will to move into that direction.
>
> I'd be very happy to participate in the work needed to implement this.
> But first we need to agree that this is where we want to go.
>
> Thanks,
>         Stefan
>