Re: [boost] To modularize, or not to modularize. What is the plan?

Subject: Re: [boost] To modularize, or not to modularize. What is the plan?
From: James E. King III (jking_at_[hidden])
Date: 2019-05-07 11:57:35

On Mon, May 6, 2019 at 7:54 PM Rene Rivera via Boost
<boost_at_[hidden]> wrote:
>
> A modular Boost, to me, means a Boost that first and foremost a collection
> of independently consumable C++ libraries.
> ...
> What would it take to reach that modular goal? Why do I keep saying we've
> been working on this for ages and ages? Briefly here's what it would take
> to get there (not in any particular order):
>
> * Abandon the single header include tree.
> * Abandon the monolithic build infrastructure.
> * Ban relative use of inter-library dependencies.
> * Explicit declaration of inter-library dependencies.
> * Strict normalized library layout.
> * Remove, and ban, dependency cycles at the inter-library user consumable
> granularity.

I would also add:

* the ability to have optional sub-modules (for example boost uuid
only depends on serialization if you want to use it)
* all direct dependencies must be automatically generated - nobody
should need to manually modify the list of dependencies for a module
(this is safe because your pull request will identify when the build
system file with the dependencies has changed, and alert you to the
fact that you added or removed a dependency).

It would be easier in the beginning if everyone agreed to target one
distribution system, such as conan, which is capable of handling the
direct dependencies, capable of downloading all direct and transitive
dependencies, and can generate additional build system details for
consuming what got downloaded (in make and cmake, and others).  For
example:

$ conan install boost_uuid/1.69.0_at_bincrafters/stable -g cmake -g txt
-g compiler_args
...
...
Generator compiler_args created conanbuildinfo.args
Generator txt created conanbuildinfo.txt
Generator cmake created conanbuildinfo.cmake

This will download a lot of stuff into ~/.conan, but the paths inside
are known only to conan.  The resulting files allow build systems to
consume what was downloaded.  I would then want to extend it further
with optional recipes so add a
boost_uuid/.../stable_optional_serialization which would bring in the
serialization and the uuid serialization header.  We could then use
conan to perform the build as well, or use another build system to do
it.

This effort would lend itself well to conversion to cmake as an actual
build system as well.  I would recommend a standalone boost-cmake
repository containing cmake scripts that all other repositories can
use for common, normalized build operations across all the
repositories.  Once this works with conan it would then be possible to
add other build systems, but I think it's important to choose one
system and get it working.  Conan has already done their own work on
packaging.We can even teach the CI jobs in Travis and Appveyor to
build and push updates into bintray for conan to deliver, for example
a fast-forward merge and push into a specially named branch like
"release/" could do all of that automatically.

This may allow simplification of the branching model, where folks
still work on develop all the time.  We would need to find a better
way to overlay local changes to other repositories on top of the
dependencies that are downloaded and managed.  With python I use a
pypi package called pypi-server which allows me to locally publish an
update to a package and then when I run tests on another module in a
venv (with tox, usually) it will pick up my locally published package
update.  This lets me stage changes that must touch more than one
repository.  Perhaps we could do something similar.  Boost could
maintain a Jfrog server we could all use for this purpose (I don't
know if JFrog offers hosted JFrog services yet).

- Jim