[boost] [tools][bcp] using boost internally in other libraries

From: Bjørn Roald (bjorn_at_[hidden])
Date: 2005-09-21 03:24:02

Hi boosters,

 

In the following I have made the assumption that it is in the spirit of the Boost license for Boost to be used as foundation for other libraries, open-source, free, commercial, or whatever.  In addition, that any modifications to the libraries is allowed for any such purpose.  However, it is my belief that most of the developers would not like to see extensive branching in the effort of providing better Boost libraries.  This would most likely stretch the potential of the C++ community in providing a better future set of standard libraries for the language.   

 

So to my main subject:

I have been working for some time to get acceptance for using Boost libraries as elements in a higher level portable platform for library development.  It has surprised me how hard it is to make compelling and attractive set of arguments for using Boost (or other libraries) without attempting to ignore some pretty serious concerns.  These concerns are:

 

- commercial support for our use of Boost

- quality assurance of Boost itself

- stability of Boost API 

- maintainability of Boost source code

- deployability of our library into environments using different versions of Boost or libraries used in boost

- configuration management issues in general for us and our customers

- platform support in Boost ( or if the platform support Boost if you like ;-) 

 

All of these concerns can be declared acceptable, given that you are willing to accept trade offs and risk.  The issue I try to raze is if it is possible to reduce the magnitude of these risks and trade offs.  

 

One of the key issues is that you have to decide on some strategy of how to maintain your boost version.  The most obvious strategy may be to stay with the latest Boost release at any time, this will however add risk as you have to adopt new changes as they are introduced into Boost and not as it fits your own release plans.  The obvious alternative is to stay with a Boost release longer, possibly until you are forced to switch. This strategy will quickly force you into maintaining your own branch as Boost developers can not be expected to support old releases.  Between these there are many shades of gray which may fit more or less well depending on your project and your customers.  If your customers themselves use boost you are likely to meet a added configuration management, testing and release related cost and risk.

 

As library developers have a very different type of customer relationship than what is the case for end application developers there are a number of concerns that make adaptation of Boost, or any other 3rd party library for that sake, more problematic for library developers.  The main issue is to what degree a library depends on other libraries and hence introduces indirect or direct dependencies to its users.  These dependencies tend to be troublesome for the library developer to support, as well as they may be troublesome for users to accept.  In fact users may easily be in situations where such dependencies are completely unacceptable.  To help to understand the issue I like to define categories of libraries which affect the degree they are affected by these concerns.

 

Cat1. open source, source distributions, allowing encapsulation

Cat2. open source, source distributions in general

Cat3. open source, binary distributions

Cat4. closed source       

 

Cat1 and Cat2 are build-able source code distributions.  This give the user the ability to build the library and ensure that it is built correctly with other library it depends on, on the target system.  Cat1 add to this the ability to hide the use of it inside other libraries, i.e. it can live concurrently in an end system with other versions of the same Cat1 library.  Cat3 and Cat4 are for this discussion as far as I can tell similar.  Cat3 may be called open source by some, but if for a variation of reasons rebuild by user is not an option, the results in this discussion becomes the same.

 

Boost is a Cat2 library.  Even if the extensive use of headers only libraries help some of the issues addressed in this post, it is not a general solution that makes Boost or part of Boost a Cat1 library. 

 

Now an example showing some dependencies:

 

MyCustomersApplication

  |---- Boost 1.33 

  |---- MyLib

          |---- Boost 1.32

 

If MyLib is Cat1 or Cat2 the chances are good that we may be ok.  Most likely MyLib will soon adopt the migration to new Boost releases by the actions of the maintainer or by patches from end users.  If however MyLib is Cat 3 or 4 we have a problem.   This problem is real even if the use of Boost in MyLib is strictly internal.  In my view this is an especially bad problem.  Even if I do not intend to use Boost in any way in the API of MyLib I still have this issue of incompatibility.  This problem will disqualify use of Boost even internally for many library developers.  Now, if we assume (which in most cases is dangerous) that library developers are a skilled race among the general population of the C++ community, it is in my view sad that many of them are not able to use Boost or potentially participate professionally in the efforts to improve Boost.  I am sure many actually would participate if they could use it at work.  Any management with commercial goals mixed with good sense of business would also let their developers participate actively, especially if this was the best way of maintaining parts of their product.  This would let their developers focus their effort on the parts of the product that is core to their business rather than the parts that can be found and downloaded on the net (read boost.org).

 

Ok, back to the example.  If MyLib have no intention of using Boost in the API, is there a way?  I have been struggling with this issue for a while.  The obvious solution to me is to make Boost a Cat1, the problem is how?  As mentioned earlier in many threads on this list header-only libraries helps, however this does not apply to all of Boost and it is possibly other issues that I am not aware of.  If we look at the C++ domain in general, there are limited ways of hiding symbols in libraries.  As far as I can see, none of these are sufficiently powerful to hide boost completely.  One alternative is however to let the symbols leak, but make sure none are in conflict with the other version of boost.  Some sort of namespace replacement or wrapping comes to my mind.

 

Side Note:  

I am aware of symbol encapsulation abilities in windows DLLs and similar new features supported by GCC, however these are not generally supported on all platforms, so I have selected to ignore them. They may however be helpful to many.   

 

The general idea is that there are no references to symbols in the boost namespace between any MyLib libraries, nor in any header files in or referenced from MyLib API.  How can this be achieved?  I have concluded that accessing boost trough namespace wrapping header files are unpractical at best.  This needs to be designed into all of boost for this to work, and I do not know exactly how.  The main challenge is #include <boost/xxxx.hpp> directives which does not refer to the wrapping headers.       

 

An alternative is to replace all references to boost namespace with e.g. MyLib::boost namespace in the boost source code and change all #include <boost/xxx.hpp> directives to #include <Mylib/boost/xxx.hpp>.  This would make a buildable library which would not share symbols with other versions of boost.  The result is a branch or clone if you like of Boost that can be used with the same effect as if we actually had a Cat1 boost library.  There is however serious problems here.  Any management with common sense would say....  hey.... stop.  The prospect of undertaking maintenance of a branch or clone of Boost does not make much sense --- So, the only way this would work is to make it possible to maintain the library in the boost community and somehow make the translation to the MyLib/boost namespace a pure production step.  In configuration management terms this means that the original or possibly slightly patched boost source code is under configuration control along with a tool-chain used under MyLib system build that transforms the boost source tree to the form used internally.  This is where I started looking at Boost tools/bcp.  

 

The bcp (boost copy I guess) tool is an excellent starting point since it already has the ability to copy selected libraries and their dependencies from the boost distribution to project specific source tree.  So in a sense it is similar in spirit, but solves primarily a different goal, which is to reduce the size of the boost distribution imported.  This is complementary to what I want.  I have experimented a little with some extensions to bcp using some simple regexp replace calls and got some encouraging results.  I am able to build the resulting source tree using only small adjustments to top level bjam files after treatment by the modified bcp.  I have yet to complete support for replacing boost with nested namespaces since it is trickier to do the replacement of the ending braces of the namespace scope.  I believe other tools than the Regexp library may be more appropriate for this task.

 

So to wrap up:

I do not believe anything prevents me from going along this path if I decide to do so.  I am however interested in the boost developers attitude toward this approach, and in particular if this is something you will encourage and/or support.  I am willing to work on the tool support for this in boost or to maintain a modified bcp tool if that is more appropriate. 

   

 

Regards

 

Bjørn Roald