Re: [boost] To modularize, or not to modularize. What is the plan?

Subject: Re: [boost] To modularize, or not to modularize. What is the plan?
From: Robert Ramey (ramey_at_[hidden])
Date: 2019-05-07 05:06:33

On 5/6/19 9:40 PM, Rene Rivera via Boost wrote:
> On Mon, May 6, 2019 at 8:17 PM Robert Ramey via Boost <boost_at_[hidden]>
> wrote:
> 
>> On 5/6/19 4:47 PM, Rene Rivera via Boost wrote:
>>> * BoostOrg would not produce a monolithic, combined, merged, etc
>>> distribution.
>>
>> I don't see it as necessary for Boost to give this up. That is, I don't
>> see the current setup as conflicting with the ability to just download
>> the libraries he wants.
>>
> 
> It conflicts in that the modular arrangement is not the same as the current
> monolithic arrangement.
> 
>> * BoostOrg would produce collectively tested milestone modular
>>> distributions.
>>
>> I don't think anything has to change here.
>>
> 
> The change would be in holding on to the belief that the single header dir
> and top level build is the face we should be putting forward to users.
> 
>> Sounds lovely right? ...I'll leave the discussion of merits to responses
>>> herein ;-)
>>
>> I envision the construction of a tool which just goes to github and
>> downloads a list of boost libraries.
> 
> 
> Can we just stop trying to build more Boost specific tools?
> 
> 
>> For each library the download
>> process makes a simple transform to a standalone directory for that
>> library.  Similar to what the global distribution currently looks like
>> except for one library at at time.
>>
> 
> You've just created an combinatorial explosion of distributions which
> you'll need to test. As users will complain when their particular
> combination doesn't work.
> 
> This would be useful right away.  In vinnies 1.70.1 situation, one could
>> put this to use right away.
> 
> 
>> a)the user changes the name of the current beast directory to beast-1.70.
>> b)downloads the latest from the master into the new beast directory.
>>
> 
> I my ideal modular view Vinnie would publish a new Beast library version of
> 1.70.1. Along with the requirement that it's can be used with its
> dependencies of 1.70.0. Users would obtain that new version with their
> existing package management method. Hence, not that different with how you
> posit, but not tied to any particular Boost custom tool. Or arrangement.
> 
>> What would it take to reach that modular goal? Why do I keep saying we've
>>> been working on this for ages and ages? Briefly here's what it would take
>>> to get there (not in any particular order):
>>>
>>> * Abandon the single header include tree.
>>> * Abandon the monolithic build infrastructure.
>>
>> One would need a more "stand alone" tool for non-header only libraries.
>>
> 
> Why? And not sure what you mean by that?
> 
> 
>> But I presume lots of users would just compile the *.cpp files into
>> their app or build their own DLL. Ideally, the library package
>> would/should contain a CMake script to do this.
>>
> 
> Yes, they could make use of whatever build system the library provides
> support for. Or they could get it from an established package manager that
> supports their method of building.
> 
>> * Ban relative use of inter-library dependencies.
>>
>> I don't think that's possible.
> 
> 
> Actually it seems we are almost there on that point. As I could only find
> one instance in the latest release of this problem. But it was a quick
> search only. So there may be more.
> 
> 
>> But I don't think that's necessary.
> 
> 
> Really? How would you resolve a library referring to another one with a
> path like "../../thread" instead of "/boost/thread" (or equivalent).
> 
> 
>> The
>> only thing is that a user would need to install the dependent libraries
>> he needs.
> 
> 
> Right, they could say, use a package manager or "manua"l downloads of what
> they need.
> 
> 
>> One could try to make a tool to do this
> 
> 
> Yes, they are called package and dependency managers. There are many of
> them.
> 
> 
>> - but I've argued
>> that that is a fools errand.
> 
> 
> Okay two things.. 1. Are you saying that package manager developers are
> fools? 2. Ah, earlier you say we should write a custom Boost tool to create
> custom distributions, and now you say it's a fools errand. Did you just
> call yourself a fool? ;-)
> 
> 
>>    Rather than argue that any more. I could
>> just imagine user does the following:
>>
>> a) Adds a boost header to his project.
>> b) "installs" that header as above
>> c) tries to build his project
>> d) If something missing - call a) for the missing thing
>>
>> at the end of that process, he has a minimal subset of boost required to
>> support his project.
>>
>> If someone has nothing else to do he could write a tool which does
>> generates a list of dependencies for a given app as a text file. Then
>> the user would do most of them in one shot.  This would likely be an
>> minor enhancement of BCP or similar program. But the result would be the
>> same.
>>
> 
> Similar tools already exist.. they are called Package & Dependency
> Managers. We should probably support them by making Boost easier to package
> by making Boost modular.
> 
> BTW - the user already incorporates non boost libraries into his project
>> using this same procedure. Ideally any dependency checking tools would
>> work on these as well.
>>
> 
> They do.
> 
>> * Explicit declaration of inter-library dependencies.
>>
>> I don't think this is necessary.
>>
> 
> Sure, it's not required. There are tools to this job for you. But they
> aren't as precise as an explicit declaration from the author. And being
> explicit helps those existing tools by making their jobs as easy as reading
> that explicit list and avoiding all the work of scanning and guessing.
> Improving everyone's lives accordingly with reduced complexity.
> 
>> * Strict normalized library layout.
>>
>> I don't think this should be necessary. But I'm aware that some
>> libraries don't follow convention regarding header layout.  So either
>> those have to change or the "downloader" tool would have to smart enough
>> to sort those out.  I don't recommend the latter option.
>>
> 
> Wait, another tool to write? Oh, wait, no.. You mean make the same tool
> even smarter. And hence prone to mistakes.
> 
>> * Remove, and ban, dependency cycles at the inter-library user consumable
>>> granularity.
>>
>> I don't think this is necessary.  If one is following a chain of headers
>> rather than a chain of modules - there are no cycles.
>>
> 
> Sure, that's a true statement given your supposition. But lets face it, the
> reality is that everyone except Robert Ramey thinks in terms of module
> granularity. All the tools we use are written in those terms. And we do it
> at the module granularity because it is useful to think about them that
> way. It simplifies things like configuring the tools we use, how we
> advertise the libraries, how we track the libraries in source control
> repositories, documentation, bug tracking, and so on.
> 
>> There's probably more items that I've forgotten above. But this should be
>>> enough to converse about.
>>
>> LOL - ya think?
>>
> 
> Haha, there's only so much time in the world.. especially when you are
> waiting for friends to arrive for usual Monday night political gathering :-)
> 
> I think you're concept of "modularized boost" is at least broadly
>> similar to mine.
>>
>> To summarize, the only things we would need:
>>
>> a) we need a tool to download/transform one boost library at a time.
>>
> 
> Those already exist. We should facilitate their existence.

Great I didn't realize that there was already a method for downloading a 
boost library an transforming the directory arrange from the boost 
develop one to the one that users use.

> b) optionally, it would be nice to have a dependency listing tool.  FYI
>> this is more difficult than it looks since the user doesn't all the
>> boost libraries on his machine.  Such a tool would have to trawl the
>> boost master on github or some oneline database of headers summaries.
>>
> 
> Those also exist. For example the Conan package manger can produce a list
> or graph of such dependencies in various format.

Double great - just pick one.
> 
> 
>> c) There would likely need to be separate directory - boost tools with a
>> couple of things in it.  Some stuff would be moved from boost root to
>> boost/tools. So the user wouldn't need the root on his machine.
>>
> 
> Yep. But likely would be minimal as we would lean on existing, external,
> tools ecosystem.
> 
> d) A good written explanation for users who want to do this.
>>
> 
> We always need documentation. There can never be enough of it!

Our problem isn't that we don't have enough documentation.  It's that we 
write tools and procedures that lack sufficient conceptual integrity to 
document.  We only discover that when we write the documentation and by 
that time it's too late to re-design.

> 
> What we don't need to do is re-organize current
>> boost/development/testing etc.  This is merely an alternative deployment
>> concept.  Boost developers would not be effected.
>>
> 
> Well... I would implore that we do not support more than one deployment
> arrangement (multiple "channels" is perfectly okay though). But a sans-root
> arrangement actually makes development easier in that it facilitates
> individual library testing. It would be considerably easier to just clone
> your one library and test it against any particular existing versions of
> other Boost libraries.. without the juggling with git submodules.

That's really an orthogonal question.  You and your team are free to 
evolve the current global/build/test in a way that you see fit. But in 
the mean time, we can just use the current one.  It doesn't need to change.

The current infrastructure is already working and in use for many years. 
It works fine for many people.  I'm suggesting an additional method of 
deployment for special cases.

> 
> That's it. just three or 4 simple things. And no disruption of current
>> setup.
>>
> 
> I feel so much better that you think it's simple.

LOL - it's simple until everyone starts mucking with it.

Basically a case like vinnies would be handled by just being able to 
produce a special version of the source that constitutes vinnie's 
library.  If we can't do that, that what would be the point of trying to 
make something more ambitious?