Subject: Re: [boost] [modularization] What is a module? What is a sub-module?
From: Rob Stewart (robertstewart_at_[hidden])
Date: 2014-09-22 05:37:30
On September 21, 2014 11:12:51 AM EDT, "BjÃ¸rn Roald" <bjorn_at_[hidden]> wrote:
>On 09/21/2014 10:36 AM, Vicente J. Botet Escriba wrote:
>> After the long threads concerning the modularization it seems clear
>> to me that we are in an impasse.
>Maybe most of the friction is more of a case of lack of clear
>communication rather than real disagreements. It could be the goals
>would be agreed if they where clear to everyone. Some participants in
>the threads seems to have clear goals in mind for what need to be done
>first, and just feel need to to proceed, while others are confused
>what is going on and why. The latter may need to understand the "why"
>in how we get to a end result we want and what that result looks like.
>The former group may be more concerned with what they "know" has to be
>done before we get anywhere. They need to convince the skeptics why
>that is the case. Neither side's statements and arguments are hard to
>understand if you are willing to try to shift mindset for the sake of
>understanding. Nevertheless it need to be some level of consensus
>before this can proceed.
You are very likely correct.
>So how can consensus be achieved? I think starting with more concrete
>meaning to terminology used in discussions, proposals and guidelines
>would be a very helpful. Guessing what people mean with module,
>sub-module, library, sub-library, repo, sub-repo, package, dependency,
>etc. is not helpful to understanding each other.
>A library is a collection of code in Boost that is reviewed and
>accepted/rejected by boost as community. A library is maintained be
>individuals that are the library maintainers. The code is managed in a
>separate git repository that is included as a git submodule in the libs
>folder of the boost master repository. A library contain the library's
>main module in subdirectories include, src, test, build, and doc. In
>addition a library may contain a number of additional directories
>containing optional modules that depend on the main module, these are
You've defined "library" in terms of "module" and "sub-library" which have not yet been defined. What is a "main module"? I need to understand that to understand what's included in a library. More on module's definition below.
>A library may contain related code in sub-libraries that should be
>treated as separate module to limit dependencies incurred if they are
>part of the library's main module. The sub-library has its own module
>structure containing its own include, src, test, build, and doc
>directories. A sub-library is part of the library and is maintained by
>the libraries maintainers.
I need to understand "module" to understand "sublibrary" (which needn't be hyphenated, BTW).
>Unit of deployment of boost source code and/or pre-build libraries,
I assume you meant "pre-built" rather than "pre-build" here.
>documentation etc. Typically there may be a one-to-one relationship
>between packages and modules, but it is possible to deploy more than
>module in a package or break one module into more than one package.
The current packaging model puts all modules into one package, so it's more than possible, it's the norm.
>A version controlled directory structure containing checked out or
>modified files in a working directory and a database of the repository
>history and relationships to other repositories. In a git working
>directory, the database is in the .git subdirectory or is pointed to by
>a .git file.
The usual meaning of "repository", at least in my experience is the managed history in a certain control tool, not the files in a workspace.
>I suggest we do not use this term mean sub-library. Use the term
>sub-library or git submodule instead.
If the VCS ever changes again, the tool-specific name of this entity will probably change. It would be better to provide an abstraction. That is, formalize "subrepository" and not that a git submodule is a subrepository.
>A organized set of boost library code that can be handled in a uniform
>manner by boost tools. A module shall contain the include, test,
>and doc directory, Modules that are not header-only shall also contain
>the src directory that is used to build one or more corresponding
How is a module distinct from a library? Both are defined in terms of the directories they contain. Each is defined in terms of the other.
>I suggest we do not use this term to mean sub-library, use sub-library
>instead. If it is not clearly given by context, use git submodule if
>have a git repository tracked using a git submodule in mind
Until I better understand the difference between "library" and "module", I can't say whether I agree with your conclusion on submodule.
>Handling of dependencies is where I struggle the most with seeing a
>clear path forward. In particular what determines the nodes and edges
>in the dependency graphs we care about. And what are we going to use
>the dependency graph for.
>Test Example, and Doc Dependencies:
>First of all, if test, example and doc code is part of the module and
>incur additional requirements, we certainly do not always want to track
>those dependencies as the modules dependencies. A separate dependency
>graph node for test code seems to be a solution if there is a real need
>to track it at all. Documentation can also clearly be treated separate
>if need be. However, given this, then the module as defined above is
>longer the node in the dependency graph. But that is probably just the
Test and doc dependencies should certainly be tracked separately, if at all.
>Modules that are not header only have source files in the src directory
>that are compiled into one or more library files (ignoring variants
>directly supported by Boost.Build). Separate dependency graph nodes may
>be appropriate here to distinguish dependencies at link and compile
>time. But there are many possible facets of this, so I think the real
>use-cases for the dependency graph should drive requirements for what
>the nodes and edges shall model. In addition dependencies may vary on
>configuration of the target environment. It is not clear if or how such
>external dependencies should be tracked, however starting with the
>Jamfile lib dependencies is certainly a good start. It may be most
>package management systems has what is needed for the rest, so it is a
>mater of bridging these worlds.
I should think dependencies would be computed at the logical grouping represented by library or module, depending on what those terms actually mean. I presume one will choose to build components by such logical entities.
>Dependencies in the include directory may cause compile and link time
>dependencies for the module user. These dependencies does not incur
>before a header is included directly or indirectly that require the
>specific dependency to be met. This could, as some have pointed out,
>be leveraged to get very flexible and fine-grained "real" dependency
>graph in boost. However, as the actual dependencies are not known
>before the application developer changes source code, compiles and
>links, and then understand cause of the resulting diagnostics, this is
>not very helpful for packaging of minimum required sub-sets of boost. I
>am also afraid the diagnostics for missing headers or object file
>symbols will not be a very user friendly solution. However if that
>could be fixed somehow to point directly at the missing package, or
>better that a package manager could be more or less automatically
>invoked to fix it, then this may be a path forward. Such fine-grained
>dependency tracking could greatly reduce need for sub-libraries.
I agree that such fine-gained tracking can be a cause of confusion and hassles. I normally prefer to think in terms of libraries, not optional features. That does less to problems managing dependencies like Date Time's optional dependency on Serialization, however.
>Separating larger chunks of code in a sub-library may seem reasonable
>for several reasons, but to separate single headers into their own
>sub-library only to get a "pretty" graph may clearly be way off the
>reasonableness scale. Especially, if it can be reasoned that we don't
>push internal boost structure problems on the helpless application
>developer to figure out. It seems reasonable to look for facilitation
>for something much simpler in these cases.
> For the lack of a better
>term for what some are suggesting, I just invented bridging-header as a
>term which may be a mechanism to help in this situations.
>A bridging header is a C++ header files that bridges facilities in one
>module with facilities in another module to provide a new convenience
>facility to users. The bridging header is part of the include
>in one of the two modules and only depend on a minimal required set of
>features from the two modules to provide the new convenience facility.
>A bridging header is marked in a to-be-determined way that allow
>dependency tracking tools to track the set of bridging headers between
>any two modules as a separate node (a bridge) in the dependency graph.
>When a user include a bridging header it add both the bridged modules
>dependencies, however it may not be practical to have every bridge
>tracked by a package manager as a separate package.
That seems like a decent approach.
(Sent from my portable computation engine)
Boost list run by bdawes at acm.org, gregod at cs.rpi.edu, cpdaniel at pacbell.net, john at johnmaddock.co.uk