Re: [boost] [modularization] Modularizing Boost (modularization)

Subject: Re: [boost] [modularization] Modularizing Boost (modularization)
From: Ahmed Charles (acharles_at_[hidden])
Date: 2013-10-25 03:41:17

> Date: Fri, 25 Oct 2013 08:24:53 +0200
> From: steveire_at_[hidden]
> To: boost_at_[hidden]
> Subject: Re: [boost] [modularization] Modularizing Boost (modularization)
> 
> > I'm still catching up on all of the replies, and I like the work overall and it seems to be going in the right direction.
> 
> Thanks for the detailed analysis.

It turns out I'm still not caught up with my reading, it seems there is a lot to say on the topic.

> > One minor concern I have with the steps you outline below is that they seem to optimizing for reducing edges in the dependency graph rather than creating the right dependency structure, which is a subtle distinction.
> 
> Yes, they are distinct. Creating the right dependency structure is more
> what I had in mind as part of phase 2.

I'd be reluctant to make lots of changes without the eventual goal in mind, but the rest of your reply seems well reasoned, so even if I don't agree on all points, things are moving in the right direction.

> >  For example, moving the exception classes used by all of boost out of the exception library doesn't really seem like a win overall, since having files grouped logically is the entire point of a module. 
> 
> I guess 'logically' might depend on point of view.
> 
> BOOST_THROW_EXCEPTION might not throw an exception, depending on the
> capabilities of the compiler, among other things. So, from my point of
> view, it can be seen as 'feature normalization' and belong in some core
> location with other parts of boost which do 'normalization' of features
> of toolkits/compilers (think most of Config and static_assert,
> type_traits etc, and someday boost.any perhaps).

I understand that perspective but after looking at the code, I still think it logically belongs in the exception library, though see below.

> The main reason I moved it out in my analysis was because the exception
> library also has other dependencies. Of course, another option would be
> to split it into two libraries to isolate those dependencies. Looking at
> the dependencies again now, even that may be unnecessary depending on
> what happens with the detail and utility repos.

I don't have the time to look right now, but I'd be curious why the exception library would have lots of dependencies and if it does, it would probably make sense to have it split into two, containing the basic functionality that every library needs and the advanced functionality that requires additional dependencies. For example, if you suggested a library called exception_core which contained the two headers above and everything using those exclusively depended on that library, I'd think it was perfect, because the files are logically placed and the dependency graph would be reduced to a usable state.

> > I think the biggest question to ask is what libraries are expected to be at the core of boost and used by all other libraries and effectively create a layering structure, where libraries belong to layers and can only depend on libraries at it's level or below. The goal is to create as many levels as possible by moving libraries up in level whenever they have no more incoming links at their current level.
> 
> That's similar to what we are doing in KDE. The kdelibs libraries in
> KDE4 are very interdependent.
> 
>  http://quickgit.kde.org/?p=kdelibs.git&a=tree&hb=master
> 
> 3 years ago I wrote "More interdependence between libraries is easier,
> but makes independent reuse much harder. We just have to find out if
> it's worth it."
> 
>  http://thread.gmane.org/gmane.comp.kde.devel.core/67458/focus=67520
> 
> and created a plan of 'tiers':
> 
>  http://techbase.kde.org/index.php?title=Projects/KDELibsModifications&oldid=55633
> 
> It took a lot of work and moving stuff around to 'clean up' kdelibs:
> 
>  http://community.kde.org/Frameworks/Epics/kdelibs_cleanups
> 
> reduce duplication with Qt provisions (note that there is duplication of
> some classes/concepts between some boost libraries) :
> 
>  http://community.kde.org/Frameworks/Epics/Reduce_class_duplication
> 
> and to do actual splitting:
> 
>  http://community.kde.org/Frameworks/Epics/Splitting_kdelibs
> 
> Today we have four split out tiers and a staging directory (libraries
> not yet ready to be moved to a tier) to manage and layer dependencies:
> 
>  http://quickgit.kde.org/?p=kdelibs.git&a=tree&hb=frameworks
> 
> and we'll soon be splitting that repo into multiple smaller git repos
> and making releases of modular 'KDE Frameworks 5'. Note that we are
> doing repo splitting after code modularization, in contrast to what
> boost is doing, and we'll use git grafts where we want the history from
> the unmodularized repo in the future.
> 
> So, yes, what boost (should be) trying to do regarding modularization
> sounds familiar to me. There are indeed several lines of work to do with
> subtle or not-so subtle distinctions between them.

I didn't actually follow all of the links, but I will soon, but just based on your comments, it seems like KDE is shaping up nicely. It seems that all the libraries (or code bases that should be libraries but aren't) in my life have the problem of being tightly coupled with developers that don't see it as a problem and are resistant to change. (Perhaps it's just human nature and it's not relevant to this thread.)

> > I also think the search should initially be focused on dependency links which make no sense, like proto -> spirit (I think that was the one Eric noticed) before moving on to dependencies which make sense but perhaps the files are in the wrong location.
> 
> Yes.
> 
> I am not familiar enough with boost though to know what makes no sense.
> I would point anyone at the dot files I linked in my original mail if
> you want to find other edges which should be removed.

I'll take a look, but I'm only really familiar with the parts of boost that are effectively standardized or on track to be and even then, not some of those (like I've never used asio). But that said, things that 'look wrong' probably stand out, even to more casual observers.

> > Lots of the changes below seem to be of the form "move utility code from library X to some library that is more heavily used than X and will therefore, reduce the graph". Is there a library that is core enough that it can be used by every other library and yet uses no other libraries? If that doesn't exist, should it?
> 
> It doesn't currently exist. I proposed creating it as part of phase 2.

It seems the config library is a good first step given that it has no dependencies, though I wouldn't want to make it stretched beyond it's scope. It seems like having a library which depends on only config and contains utility-esque code would be helpful (I'm assuming it will have to be created rather than an existing library fulfilling its place, unless the detail library gets rearranged significantly).

> > Wouldn't it make more sense for iterator and iterator_traits to be in the iterator library? I can imagine moving lots of useful utilities to the 'detail' library simply cause that's what it's for, but the type_traits library shouldn't be used to contain iterator related code that has nothing to do with type_traits.
> 
> The motivation here is similar to what I wrote above about exceptions.
> Look at the dependencies of the iterator library in graph_all.dot to see
> how it contributes to the graph.
> 
> Note also that boost/detail/iterator.hpp actually only contains traits
> helpers. So both files are about iterator traits. Moving them is
> appropriate in my point of view because they are related to traits, and
> because most users of those files do not need most of the iterator
> library or its dependencies.

I almost buy this argument, but I suppose I'm still on the fence. If it's only traits, perhaps a better place would be that mythical library with no dependencies other than config. Ideally, type_traits is also something that is less of an issue when C++11/14 come into play.

> > This makes sense. Though it seems like you are suggesting that config is more widely used than detail, which seems strange given detail's goal as being the melting pot of various little things without a home. It's not obvious to me why moving something from one to the other really makes a big difference, but version code should be in a library that has no dependencies, like the library I suggested above.
> 
> Config also has no dependencies (as of
> https://svn.boost.org/trac/boost/changeset/85274 )
> 
> But you wrote and as I wrote in the OP, there may be sense in creating a
> core library/repo of which config is a part.

That is beginning to seem obvious.

> > This makes no sense to me, why would property_map code belong in the graph library? They seem to be solving completely different problems and my naive assumption would be that they have no dependencies on each other, though if that were the case, we probably wouldn't have a dependency graph that looks like spagetti. Anyways, perhaps you could explain this in more detail?
> 
> I think Edward and Jeremiah have made progress on this issue, though I
> admit I didn't follow the details closely, so I can't say more:
> 
>  http://thread.gmane.org/gmane.comp.lib.boost.devel/245078/focus=245276

Hopefully one of them can comment.

> > Thanks again for doing all this research. It's definitely useful and in the right direction.
> 
> And thanks for the feedback!

It's my pleasure. As a user of boost, I'd love to see it more modular and have a sane dependency graph.