[boost] [modularization] Modularizing Boost (modularization)

Subject: [boost] [modularization] Modularizing Boost (modularization)
From: Stephen Kelly (steveire_at_[hidden])
Date: 2013-10-17 18:24:57

Hi there,

my plan for modularizing and modernizing Boost was roughly this:

* Phase 0 - remove dead weight by bumping compiler feature requirements
* Phase 1 - move some files around so that the modularized repos form a
mostly directed graph
* Phase 2 - Form some kind of 'boost core library' or 'boost feature
normalization library' from the guts of existing libraries like
type_traits, static_assert, config mpl and utilities.
* Phase 3 - Try to port the mpl to variadic templates so that the
dependency on Boost.PP is not needed when variadic templates are available.

I recommend completing up to phase 2 before migrating to git, as
otherwise files will have to be deleted from one repo, and added to
another without history.

Phase 0 is aborted, so let's look at phase 1.

[Note: This should go without saying, but my experience on this list
tells me it needs to be said:

* I do not propose making many commits with the commit message 'migrate'
* The below is scripted for your understanding only. Scripting gives an
exactness which prose does not, and allows reproducibility. Don't take
it too literally.
* Of course, forwarding headers should be understood to be left behind
when a file is moved, where possible.
* Don't cry too loudly about where I'm moving files to. See Phase 2
above, and understand that this is partly only an experiment to see how
modularization can be done.

EndNote]

If the dependencies between repositories are analysed, the result is
this graph:

 http://steveire.com/boost/graph_all.dot
 http://steveire.com/boost/graph_all_small.png

There are 104 nodes and 1159 edges. Each edge is a real dependency which
exists between repos, and which will exist between boost modularized
packages. They should not be considered optional.

If we remove any nodes which are not strongly connected, we are left
with a new graph of nodes which are all strongly connected:

 http://steveire.com/boost/graph_strong.dot
 http://steveire.com/boost/graph_strong_small.png

There are 68 nodes and 675 edges.

Move enable_if from utility to type_traits:

  git submodule foreach 'git grep -l -P utility/enable_if.hpp --
include/boost | xargs sed -i
"s|utility/enable_if.hpp|type_traits/enable_if.hpp|" || echo'
  mv libs/utility/include/boost/utility/enable_if.hpp
libs/type_traits/include/boost/type_traits
  git submodule foreach 'git add . || echo'
  git submodule foreach 'git commit -am migrate || echo'

There are now 670 edges and 68 nodes.

Next, move boost/detail/workaround.hpp to the config library/repo:

  git submodule foreach 'git grep -l -P boost/detail/workaround.hpp --
include/boost | xargs sed -i
"s|boost/detail/workaround.hpp|boost/config/workaround.hpp|" || echo'
  mv libs/detail/include/boost/detail/workaround.hpp
libs/config/include/boost/config/
  git submodule foreach 'git add . || echo'
  git submodule foreach 'git commit -am migrate || echo'

There are now 661 edges and 68 nodes.

Next, move boost/limits.hpp to the config library/repo:

  git submodule foreach 'git grep -l -P boost/limits.hpp --
include/boost | xargs sed -i
"s|boost/limits.hpp|boost/config/limits.hpp|" || echo'
  mv libs/detail/include/boost/limits.hpp libs/config/include/boost/
  git submodule foreach 'git add . || echo'
  git submodule foreach 'git commit -am migrate || echo'

There are now 653 edges and 68 nodes.

Next, move boost/detail/iterator.hpp and
boost/iterator/iterator_traits.hpp to the type_traits library/repo:

  git submodule foreach 'git grep -l -P boost/detail/iterator.hpp --
include/boost | xargs sed -i
"s|boost/detail/iterator.hpp|boost/type_traits/detail/iterator.hpp|" ||
echo'
  git submodule foreach 'git grep -l -P
boost/iterator/iterator_traits.hpp -- include/boost | xargs sed -i
"s|boost/iterator/iterator_traits.hpp|boost/type_traits/iterator_traits.hpp|"
|| echo'
  mv libs/detail/include/boost/detail/iterator.hpp
libs/type_traits/include/boost/type_traits/detail/
  mv libs/iterator/include/boost/iterator/iterator_traits.hpp
libs/type_traits/include/boost/type_traits
  git submodule foreach 'git add . || echo'
  git submodule foreach 'git commit -am migrate || echo'

There are now 650 edges and 68 nodes.

Next, move boost/iterator.hpp from iterator to the utility library/repo:

  mv libs/iterator/include/boost/iterator.hpp libs/utility/include/boost/
  git submodule foreach 'git add . || echo'
  git submodule foreach 'git commit -am migrate || echo'

There are now 649 edges and 68 nodes. utility no longer depends on iterator.

Move boost/version.hpp to config:

  mv libs/detail/include/boost/version.hpp libs/config/include/boost/
  git submodule foreach 'git add . || echo'
  git submodule foreach 'git commit -am migrate || echo'

There are now 512 edges and 64 nodes.
The config, integer, io and static_assert libraries are no longer part
of the mesh

Next, move boost/pointee.hpp from iterator to detail:

  mv ./libs/iterator/include/boost/pointee.hpp libs/detail/include/boost/
  git submodule foreach 'git add . || echo'
  git submodule foreach 'git commit -am migrate || echo'

Move exception/detail/attribute_noreturn.hpp into config:

  git submodule foreach 'git grep -l -P
boost/exception/detail/attribute_noreturn.hpp -- include/boost | xargs
sed -i
"s|boost/exception/detail/attribute_noreturn.hpp|boost/config/attribute_noreturn.hpp|"
|| echo'
  mv
libs/exception/include/boost/exception/detail/attribute_noreturn.hpp
libs/config/include/boost/config/
  git submodule foreach 'git add . || echo'
  git submodule foreach 'git commit -am migrate || echo'

Move boost/throw_exception.hpp and exception.hpp into utility:

  mv libs/exception/include/boost/throw_exception.hpp
libs/utility/include/boost
  mv libs/exception/include/boost/exception/exception.hpp
libs/utility/include/boost/utility
  git submodule foreach 'git grep -l -P boost/exception/exception.hpp --
include/boost | xargs sed -i
"s|boost/exception/exception.hpp|boost/utility/exception.hpp|" || echo'
  git submodule foreach 'git add . || echo'
  git submodule foreach 'git commit -am migrate || echo'

There are now 485 edges and 64 nodes. Several libraries form an inner mesh:

 $ grep -P "^\s*(type_traits|mpl|detail|utility|smart_ptr|typeof)"
output.dot
  detail->mpl
  detail->type_traits
  detail->smart_ptr
  detail->utility
  utility->mpl
  utility->type_traits
  utility->detail
  type_traits->detail
  type_traits->mpl
  type_traits->typeof
  type_traits->utility
  smart_ptr->type_traits
  smart_ptr->detail
  smart_ptr->utility
  mpl->type_traits
  mpl->detail
  mpl->utility
  typeof->mpl
  typeof->type_traits

If we treat them as one element for now, we get a new graph from the rest

 http://steveire.com/boost/graph_with_corelib.dot
 http://steveire.com/boost/graph_with_corelib.png

There are now 138 edges and 39 nodes

Move parts of {vector_,}property_map into graph_parallel:

  sed '1,/BOOST_GRAPH_USE_MPI/d;/BOOST_GRAPH_USE_MPI/,$d'
libs/property_map/include/boost/property_map/property_map.hpp > tmp.hpp
  sed -i '/BOOST_GRAPH_USE_MPI/,/BOOST_GRAPH_USE_MPI/d'
libs/property_map/include/boost/property_map/property_map.hpp
  sed '1,/BOOST_GRAPH_USE_MPI/d;/BOOST_GRAPH_USE_MPI/,$d'
libs/property_map/include/boost/property_map/vector_property_map.hpp >>
tmp.hpp
  sed -i '/BOOST_GRAPH_USE_MPI/,/BOOST_GRAPH_USE_MPI/d'
libs/property_map/include/boost/property_map/vector_property_map.hpp
  sed -i '/#undef PBGL_DISTRIB_PMAP/r tmp.hpp'
libs/property_map/include/boost/property_map/parallel/distributed_property_map.hpp
  rm tmp.hpp

  git submodule foreach 'git grep -l -P
boost/property_map/parallel/distributed_property_map.hpp --
include/boost | xargs sed -i
"s|boost/property_map/parallel/distributed_property_map.hpp|boost/graph/parallel/distributed_property_map.hpp|"
|| echo'
  git submodule foreach 'git grep -l -P
boost/property_map/parallel/impl/distributed_property_map.ipp --
include/boost | xargs sed -i
"s|boost/property_map/parallel/impl/distributed_property_map.hpp|boost/graph/parallel/detail/distributed_property_map.hpp|"
|| echo'
  mv
libs/property_map/include/boost/property_map/parallel/distributed_property_map.hpp
libs/graph_parallel/include/boost/graph/parallel
  mv
libs/property_map/include/boost/property_map/parallel/impl/distributed_property_map.ipp
libs/graph_parallel/include/boost/graph/parallel/detail
  git submodule foreach 'git add . || echo'
  git submodule foreach 'git commit -am migrate || echo'

 http://steveire.com/boost/graph_after_parallel.dot
 http://steveire.com/boost/graph_after_parallel.png

There are now 95 edges and 33 nodes

The remaining problematic edges are:

 conversion->range
 conversion->math
 range->algorithm
 math->multiprecision
 concept_check->parameter

Assuming they can be broken by moving some files aroung (I believe they
can be), we end up with a small graph of strongly connected components:

 http://steveire.com/boost/graph_after_remaining.dot
 http://steveire.com/boost/graph_after_remaining.png

There are now 18 edges and 11 nodes.

Looking at the entire graph again, we get this:

 http://steveire.com/boost/graph_final.dot
 http://steveire.com/boost/graph_final_small.png

Obviously, this is not perfect, but it is a beginning, and it is mostly
now a directed graph.

Any comments?

Thanks,

Steve.