Re: [boost] proposal - modularize Boost build system

Subject: Re: [boost] proposal - modularize Boost build system
From: Robert Ramey (ramey_at_[hidden])
Date: 2017-06-18 17:33:57

On 6/18/17 9:09 AM, Stefan Seefeld via Boost wrote:
> One of Boost's bottlenecks is its monolithic build infrastructure. It
> would be beneficial for many different use cases (testing, packaging,
> etc.) to decouple the building of Boost libraries (well, those that
> require building, see the listing in
> http://www.boost.org/doc/libs/1_64_0/more/getting_started/unix-variants.html#header-only-libraries)
> such that they may optionally be built stand-alone.
> 
> At present, building a Boost library requires the entire
> (super-)repository to be checked out, and the build logic itself
> involves traversing the entire source tree, looking for a "Jamroot".
> 
> What would be beneficial to many would be a workflow like this:
> 
> 1) Have a development environment with (some version of) Boost
> pre-installed (or at least the parts that the library being built
> depends on).

sort of a Boost without the libraries - a Boost developers kit?

> 2) Check out a single Boost repository (e.g.,
> https://github.com/boostorg/python)
> 
> 3) Invoke a command to build it (if there is anything to build)
> 
> 4) Invoke a command to test it
> 
> 5) Invoke a command to install it
Hmmm - I don't know what this would actually do than just copy the built 
library to some "special place"
> 
> 6) Invoke a command to package it (optional)
After all this time I have no idea what "packaging" means in this 
context. I never could figure out what CPack is for or what it would do 
for me.

> While it's of course already possible to do all the above by adding
> support for another build infrastructure (point in case: Boost.Python
> right now uses SCons for this very reason), this means duplication of
> effort, as Boost as a whole is still built, tested (and even packaged
> for binary packages) using Boost.Build, meaning I need to maintain two
> sets of build infrastructure.
> 
> This proposal thus has two parts:
> 
> 1) Make the nested build logic independent of the outer bits, so
> individual libraries can be built stand-alone (for example, using b2 by
> adding a `--with-boost` set of options to point to the location of the
> prerequisite parts of Boost).

Hmmm - too fuzzy.  For me, I just move to the library directory test and 
invoke b2 and it builds and runs the tests.  Then I just copy the built 
library from the bin.v2 tree to whereever I want so I'm done.  I'm 
thinking this is addressed in the current system.

> 2) Define a clear interface the outer build logic will use to invoke the
> nested build commands.

too fuzzy.  Needs to be more specific.

> Note that for my own case above (Boost.Python), providing 2) would be
> enough, i.e. by having a way for me to "plug in" my SCons commands to
> build and test Boost.Python could obsolete the existing Boost.Build
> logic as defined in Boost.Python (as well as the prerequisite parts in
> Boost.Build's "python" module).

In practice I invoke b2 with my own shell script which builds/runs the 
tests and generates my own cool test matrix - library_status. Doesn't 
that mean that a library/build/test can't be done from just about any 
system including windows IDE, CMake etc.

> However, as my proposal is *not* actually advocating to move away from
> Boost.Build, but rather to modularize it, I think 1) is essential to let
> everyone else (who may not be inclined to use anything other than
> Boost.Build) to also take advantage of modularization.

> While I'd rather avoid delving into the technical details of how this
> could possibly be implemented (if you really must, *please* do so in a
> new thread !), let me outline a few use-cases that would be made
> possible by the above:
> 
> 
> * individual projects would be free to switch to their preferred
> infrastructure, including CMake, SCons, etc., if they so wish
> 
> * individual projects could be much more easily developed and contributed to
> 
> * individual projects could be much more easily tested, notably in CI
> environments
> 
> * individual projects could be much more easily packaged
> 
> All of the above advantages are *huge*, and reflect real-world needs,
> and the technical issues to solve these are all minor. The question
> really is whether there is enough will to move into that direction.
> 
> I'd be very happy to participate in the work needed to implement this.

LOL - you'll regret this.

> But first we need to agree that this is where we want to go.

LOL - way too ambitious.  How about suggestion some incremental 
enhancements which would support decoupling.  Example

Systems which want to support CMake build should have a CMakeList.txt 
script in the library root.  Systems which support Bjam Build should 
have a Jamfile script in the library root, etc.  There is no reason that 
a library cannot support more than one build system if the library 
maintainer is willing to do the work.

Bjam in the boost root would invoke bjam on on the libraries which 
support bjam.  CMakeList.txt in the boost root would invoke build/test 
for all libraries with CMakeList.txt in the root.  Bjam/CMake would 
inhabit parallel universes - perhaps both simultaneously. So boost would 
be in effect a practical application of ideas developed in the area of 
quantum mechanics.  For now, Bjam would be required - others would be 
optional. In the future, this requirement might be relaxed.

FWIW I actually need both Bjam and CMake.  CMake can only do a half 
assed test/build of the serialization library.  BUT only CMake can build 
my Xcode IDE project.  An xcode project has at least 1000 settings.

All in all I like this proposal and approach.  It will let CMake 
enthusiasts do their thing to create something that is useful to me and 
users without making my life miserable. I'm actually skeptical that they 
can actually do this, but this doesn't prevent me from rooting for them.

Robert Ramey