Re: [boost] To modularize, or not to modularize. What is the plan?

Subject: Re: [boost] To modularize, or not to modularize. What is the plan?
From: Rene Rivera (grafikrobot_at_[hidden])
Date: 2019-05-07 04:40:05

On Mon, May 6, 2019 at 8:17 PM Robert Ramey via Boost <boost_at_[hidden]>
wrote:

> On 5/6/19 4:47 PM, Rene Rivera via Boost wrote:
> > * BoostOrg would not produce a monolithic, combined, merged, etc
> > distribution.
>
> I don't see it as necessary for Boost to give this up. That is, I don't
> see the current setup as conflicting with the ability to just download
> the libraries he wants.
>

It conflicts in that the modular arrangement is not the same as the current
monolithic arrangement.

> * BoostOrg would produce collectively tested milestone modular
> > distributions.
>
> I don't think anything has to change here.
>

The change would be in holding on to the belief that the single header dir
and top level build is the face we should be putting forward to users.

> Sounds lovely right? ...I'll leave the discussion of merits to responses
> > herein ;-)
>
> I envision the construction of a tool which just goes to github and
> downloads a list of boost libraries.

Can we just stop trying to build more Boost specific tools?

> For each library the download
> process makes a simple transform to a standalone directory for that
> library.  Similar to what the global distribution currently looks like
> except for one library at at time.
>

You've just created an combinatorial explosion of distributions which
you'll need to test. As users will complain when their particular
combination doesn't work.

This would be useful right away.  In vinnies 1.70.1 situation, one could
> put this to use right away.

> a)the user changes the name of the current beast directory to beast-1.70.
> b)downloads the latest from the master into the new beast directory.
>

I my ideal modular view Vinnie would publish a new Beast library version of
1.70.1. Along with the requirement that it's can be used with its
dependencies of 1.70.0. Users would obtain that new version with their
existing package management method. Hence, not that different with how you
posit, but not tied to any particular Boost custom tool. Or arrangement.

> What would it take to reach that modular goal? Why do I keep saying we've
> > been working on this for ages and ages? Briefly here's what it would take
> > to get there (not in any particular order):
> >
> > * Abandon the single header include tree.
> > * Abandon the monolithic build infrastructure.
>
> One would need a more "stand alone" tool for non-header only libraries.
>

Why? And not sure what you mean by that?

> But I presume lots of users would just compile the *.cpp files into
> their app or build their own DLL. Ideally, the library package
> would/should contain a CMake script to do this.
>

Yes, they could make use of whatever build system the library provides
support for. Or they could get it from an established package manager that
supports their method of building.

> * Ban relative use of inter-library dependencies.
>
> I don't think that's possible.

Actually it seems we are almost there on that point. As I could only find
one instance in the latest release of this problem. But it was a quick
search only. So there may be more.

> But I don't think that's necessary.

Really? How would you resolve a library referring to another one with a
path like "../../thread" instead of "/boost/thread" (or equivalent).

> The
> only thing is that a user would need to install the dependent libraries
> he needs.

Right, they could say, use a package manager or "manua"l downloads of what
they need.

> One could try to make a tool to do this

Yes, they are called package and dependency managers. There are many of
them.

> - but I've argued
> that that is a fools errand.

Okay two things.. 1. Are you saying that package manager developers are
fools? 2. Ah, earlier you say we should write a custom Boost tool to create
custom distributions, and now you say it's a fools errand. Did you just
call yourself a fool? ;-)

>   Rather than argue that any more. I could
> just imagine user does the following:
>
> a) Adds a boost header to his project.
> b) "installs" that header as above
> c) tries to build his project
> d) If something missing - call a) for the missing thing
>
> at the end of that process, he has a minimal subset of boost required to
> support his project.
>
> If someone has nothing else to do he could write a tool which does
> generates a list of dependencies for a given app as a text file. Then
> the user would do most of them in one shot.  This would likely be an
> minor enhancement of BCP or similar program. But the result would be the
> same.
>

Similar tools already exist.. they are called Package & Dependency
Managers. We should probably support them by making Boost easier to package
by making Boost modular.

BTW - the user already incorporates non boost libraries into his project
> using this same procedure. Ideally any dependency checking tools would
> work on these as well.
>

They do.

> * Explicit declaration of inter-library dependencies.
>
> I don't think this is necessary.
>

Sure, it's not required. There are tools to this job for you. But they
aren't as precise as an explicit declaration from the author. And being
explicit helps those existing tools by making their jobs as easy as reading
that explicit list and avoiding all the work of scanning and guessing.
Improving everyone's lives accordingly with reduced complexity.

> * Strict normalized library layout.
>
> I don't think this should be necessary. But I'm aware that some
> libraries don't follow convention regarding header layout.  So either
> those have to change or the "downloader" tool would have to smart enough
> to sort those out.  I don't recommend the latter option.
>

Wait, another tool to write? Oh, wait, no.. You mean make the same tool
even smarter. And hence prone to mistakes.

> * Remove, and ban, dependency cycles at the inter-library user consumable
> > granularity.
>
> I don't think this is necessary.  If one is following a chain of headers
> rather than a chain of modules - there are no cycles.
>

Sure, that's a true statement given your supposition. But lets face it, the
reality is that everyone except Robert Ramey thinks in terms of module
granularity. All the tools we use are written in those terms. And we do it
at the module granularity because it is useful to think about them that
way. It simplifies things like configuring the tools we use, how we
advertise the libraries, how we track the libraries in source control
repositories, documentation, bug tracking, and so on.

> There's probably more items that I've forgotten above. But this should be
> > enough to converse about.
>
> LOL - ya think?
>

Haha, there's only so much time in the world.. especially when you are
waiting for friends to arrive for usual Monday night political gathering :-)

I think you're concept of "modularized boost" is at least broadly
> similar to mine.
>
> To summarize, the only things we would need:
>
> a) we need a tool to download/transform one boost library at a time.
>

Those already exist. We should facilitate their existence.

b) optionally, it would be nice to have a dependency listing tool.  FYI
> this is more difficult than it looks since the user doesn't all the
> boost libraries on his machine.  Such a tool would have to trawl the
> boost master on github or some oneline database of headers summaries.
>

Those also exist. For example the Conan package manger can produce a list
or graph of such dependencies in various format.

> c) There would likely need to be separate directory - boost tools with a
> couple of things in it.  Some stuff would be moved from boost root to
> boost/tools. So the user wouldn't need the root on his machine.
>

Yep. But likely would be minimal as we would lean on existing, external,
tools ecosystem.

d) A good written explanation for users who want to do this.
>

We always need documentation. There can never be enough of it!

What we don't need to do is re-organize current
> boost/development/testing etc.  This is merely an alternative deployment
> concept.  Boost developers would not be effected.
>

Well... I would implore that we do not support more than one deployment
arrangement (multiple "channels" is perfectly okay though). But a sans-root
arrangement actually makes development easier in that it facilitates
individual library testing. It would be considerably easier to just clone
your one library and test it against any particular existing versions of
other Boost libraries.. without the juggling with git submodules.

That's it. just three or 4 simple things. And no disruption of current
> setup.
>

I feel so much better that you think it's simple.

-- 
-- Rene Rivera
-- Grafik - Don't Assume Anything
-- Robot Dreams - http://robot-dreams.net