Re: [boost] [svn/git/hg] Support for modularization of Boost?

Subject: Re: [boost] [svn/git/hg] Support for modularization of Boost?
From: Bjørn Roald (bjorn_at_[hidden])
Date: 2012-04-04 04:56:52

On 04/04/2012 09:20 AM, Robert Ramey wrote:
> Beman Dawes wrote:
>> As far as I can see, scaling Boost up to a much larger number of
>> libraries implies decentralization and decoupling, probably in the
>> form of per-library modules or something similar.
> 100 % aggree with this.
>
>> Modularization seems to have been missed in the discussions of
>> Subversion, Git, and Mercurial. Do distributed version control systems
>> in general and Git in particular have any important
>> advantages/disadvantages over svn for highly modularized projects?
> Hmmm - if boost is "modularized" - wouldn't that mean that
> each library has it's own repository.

I think this is a good idea as a basic assumption.  Further I think an 
official repository should be available somewhere well known for each 
module from where the integration process can pull the changes.

> Or that each developer
> can decide where he want's to store his version?

maybe
> I would think that the testing/deployment process could
> update/export each library from a different place without
> much problem.

Not very clear what testing/deployment process mean here, but I think 
this can work fine.

However, I think there also should be simple ways of using subset of 
modules from boost without fetching them from a release of the complete 
thing.  Also, it should be simple to patch with per module hot-fixes 
that never makes it into a main boost dot release. This implies support 
for per module dependency control with explicit declaration of versions 
of the dependencies supported with each version of a module.

Stuff like git submodule may provide part of the solution to this, but 
there are issues that need to be resolved such as the fact that 
currently a boost library does not fall into a a single mount point in 
the boost structure.  Typically there is at least 3:

* lib/x/
* boost/x.hpp
* boost/x/

This is something git-submodule does not handle well out of the box, at 
least I think so.  We have been down this path before on the list, I 
think possible appealing alternatives are something like:

* SCM tool support multiple/overlapping mount points     [1]
* SCM tool stage header files to common -I area         [1][2][3]
* Build tool stage header files to common -I area        [2][3][4]
* Move lib/x/.. files to boost/x/lib/... 
                                               [5]

A less appealing option is:

* clients use multiple -I when building against boost modules that has 
not been integrated

A related issue that need to be considered is the need for common 
baselines for dependencies, or at least some scheme for resolving valid 
combinations of module versions (e.g. git commits).  In boost there are 
likely to multiple dependency paths to the same modules.  How shall the 
tools and processes support this when modules not pipelined through a 
release process and aligned there?  Assuming there will be such 
alignment process for boost releases, but not for assuring compatibility 
of more ad-hoc combinations of module versions.

One thought that come to my mind is that Boost need to track which 
combination of module versions are tested in the test and release 
process.  Maybe tool support for this tracking also can track 
compatibility of other module combinations not provided by the official 
boost release team.  I guess in git this is simply a set of module names 
v.s. SHA1 keys similar to what git submodule uses.  If there is a way to 
publish results of testing other module combinations and all these 
results are combined somehow into a sort of compatibility graph, this 
may be a good way of sharing test results.

notes:
[1]
It should be simple enough for "possibly modified" git-submodule to 
support this in a naive solution.  However this may open the door for an 
evil twin like problem that require some management.  Evil twin is a 
term used IBM to denote when multiple elements (files) under SCM control 
map to the same file system path and one element eclipse (hides) the 
other, see http://www-01.ibm.com/support/docview.wss?uid=swg21125072

[2]
This requires that header module files for module in boost/... are moved 
to lib/x/boost/... or something similar

[3]
Make sure the staged header files are links to the files under SCM 
control.  Copied files are a real annoyance during debug - fix - debug 
cycles, believe me!  So whatever tool that does such staging in a 
development environment should use links whenever possible, preferably 
hard links to files or directory links (symbolic). Only when this is not 
supported on the file system should the tool use copy.

[4]
evil twins are still a problem, but now a build time evil overwrite issue.

[5]
For this to be viable, boost/x.hpp files need to be in some common 
module.  I guess this will not be bad as long as they are forwarding 
header and dependency from module x to common module is explicit end 
version controlled.

-- 
Bjørn