Re: [boost] proposal - modularize Boost build system

Subject: Re: [boost] proposal - modularize Boost build system
From: Edward Diener (eldiener_at_[hidden])
Date: 2017-06-18 20:34:26

On 6/18/2017 12:09 PM, Stefan Seefeld via Boost wrote:
> One of Boost's bottlenecks is its monolithic build infrastructure. It
> would be beneficial for many different use cases (testing, packaging,
> etc.) to decouple the building of Boost libraries (well, those that
> require building, see the listing in
> http://www.boost.org/doc/libs/1_64_0/more/getting_started/unix-variants.html#header-only-libraries)
> such that they may optionally be built stand-alone.
> 
> At present, building a Boost library requires the entire
> (super-)repository to be checked out, and the build logic itself
> involves traversing the entire source tree, looking for a "Jamroot".
> 
> What would be beneficial to many would be a workflow like this:
> 
> 1) Have a development environment with (some version of) Boost
> pre-installed (or at least the parts that the library being built
> depends on).
> 
> 2) Check out a single Boost repository (e.g.,
> https://github.com/boostorg/python)
> 
> 3) Invoke a command to build it (if there is anything to build)
> 
> 4) Invoke a command to test it
> 
> 5) Invoke a command to install it
> 
> 6) Invoke a command to package it (optional)
> 
> 
> While it's of course already possible to do all the above by adding
> support for another build infrastructure (point in case: Boost.Python
> right now uses SCons for this very reason), this means duplication of
> effort, as Boost as a whole is still built, tested (and even packaged
> for binary packages) using Boost.Build, meaning I need to maintain two
> sets of build infrastructure.
> 
> This proposal thus has two parts:
> 
> 1) Make the nested build logic independent of the outer bits, so
> individual libraries can be built stand-alone (for example, using b2 by
> adding a `--with-boost` set of options to point to the location of the
> prerequisite parts of Boost).
> 
> 2) Define a clear interface the outer build logic will use to invoke the
> nested build commands.
> 
> 
> Note that for my own case above (Boost.Python), providing 2) would be
> enough, i.e. by having a way for me to "plug in" my SCons commands to
> build and test Boost.Python could obsolete the existing Boost.Build
> logic as defined in Boost.Python (as well as the prerequisite parts in
> Boost.Build's "python" module).
> 
> However, as my proposal is *not* actually advocating to move away from
> Boost.Build, but rather to modularize it, I think 1) is essential to let
> everyone else (who may not be inclined to use anything other than
> Boost.Build) to also take advantage of modularization.
> 
> 
> While I'd rather avoid delving into the technical details of how this
> could possibly be implemented (if you really must, *please* do so in a
> new thread !), let me outline a few use-cases that would be made
> possible by the above:
> 
> 
> * individual projects would be free to switch to their preferred
> infrastructure, including CMake, SCons, etc., if they so wish
> 
> * individual projects could be much more easily developed and contributed to
> 
> * individual projects could be much more easily tested, notably in CI
> environments
> 
> * individual projects could be much more easily packaged
> 
> 
> All of the above advantages are *huge*, and reflect real-world needs,
> and the technical issues to solve these are all minor. The question
> really is whether there is enough will to move into that direction.
> 
> I'd be very happy to participate in the work needed to implement this.
> But first we need to agree that this is where we want to go.

A serious problem to consider, whenever anyone speaks of a modularized 
Boost where a library can be distributed on its own, is library 
dependencies. This problem exists for all libraries, not just Boost. But 
most libraries have their own dependency system, usually based on 
whatever OS that library is being used, whereas Boost libraries almost 
always intend to be cross-platform. How should Boost solve this problem 
is for me the bottleneck of distributing a particular Boost library.

When I speak of dependencies I am not just speaking of library X 
depending on other libraries A, B, and C etc. I am also speaking of 
library X depending on particular versions of library A, B, C etc. But 
since the only versioning system Boost has is a single version number 
for a Boost release, and since their is no way a library can check even 
that single version number of a Boost release either at compile or 
run-time, Boost libraries have no way to check versioning of other 
individual Boost libraries on which a library may depend.

If you say, I am going to distribute my library X with particular 
releases of library A, B, and C etc. with which I know my library X will 
work correctly you then have end-users of your library who may have 
multiple copies of libraries X, A, B, and C etc. on their systems, 
running each set within its own environment, not knowing how to identify 
each set of libraries, and hoping they can run these things and avoid 
the well-known shared library hell which has plagued end-users for 
years. Let's be realistic, this is a real problem which only some sort 
of Boost individual library versioning system for starters can hope to 
solve.

> 
> Thanks,
>          Stefan
>