Re: [boost] [tools][bcp] using boost internally in other libraries

From: bjorn_at_[hidden]
Date: 2005-09-24 04:05:33

On Saturday 24 September 2005 01:09, David Abrahams wrote:
> Bjørn Roald <bjorn_at_[hidden]> writes:
> > I have been working for some time to get acceptance for using Boost
> > libraries as elements in a higher level portable platform for library
> > development.  It has surprised me how hard it is to make compelling
> > and attractive set of arguments for using Boost (or other libraries)
> > without attempting to ignore some pretty serious concerns.  These
> > concerns are:
> >
> > - commercial support for our use of Boost
>
> I know of a certain company offering commercial support for boost :)

Yes, I know about Boost Consulting.  I did not mean to insinuate that there 
are no comercial support for Boost, just that it for many managers is 
important to have this kind of arrangement as part of a risk midigation in 
relation to open source.  I expected you to react on the above, but left it 
anyway for completness, it also highlights that what you are doing is 
important.  Also in face of licency issues it seems like it is hard for many 
to trust that you really are allowed to do what you are allowed to do with 
Boost or similar licences, for those none-beleivers it becommes easier to 
accept it if they acctually buy a product from somebody.

I have no first hand experience in dealing with Boost Consulting, nether do I 
know anybody that have evaluated the company and their services.  So I am in 
no possition to have an opinion about the quallity of what you can offer.  It 
is certainly a positive sign that the 3 of you are so active on the 
developers list :-)

>
> > - quality assurance of Boost itself
>
> Specifically?

Again, and as I mentioned to John, there are allways room for improvement, but 
I do think the boost developers do a lot.  I bring this up because it is a 
genuine concern whenever you deal with sub-contractors or whatever, and it 
also apply to use of open source.  

Boost and or Boost Consulting could possibly make a more prominent reference 
to QA efforts and procedures on their web sites.  At least have a bullet for 
it an tie together the procedures you have with peer reviews, testing, 
regression testing, bug tracking, change management, public developer mailing 
list.  Then you can call this web page and its guidelines your Quallity 
Assuruance Plan.

>
> > - stability of Boost API
>
> Yeah, there are no rules about it.  However, we generally avoid
> breaking source compatibility.

If the library is good enough, this is a risk many will choose to live with.  
But it would be nice if library maintainers made ther _current_ view of how 
stable the API was public.  Maybe on a scale used for all Boost libraries.

>
> > - maintainability of Boost source code
>
> I actually find that most of the time it's pretty easy to dig into a
> Boost library and fix things, 

good

> although I seldom have to because the 
> authors tend to be responsive to bug reports.  

I have noticed that

> However, there are no 
> uniform standards for certain things that make maintenance easier for
> some people, like code formatting.  But you probably had something
> else in mind?

Well I am pragmatic, we (programmers) will never agree on that stuff 
anyway ;-)

>
> > - deployability of our library into environments using different
> >   versions of Boost or libraries used in boost
>
> Hmm, that's a tough one, especially if your library traffics in boost
> types in its public API, or if you use compilers that don't support
> symbol visibility control.

yes

>
> > - configuration management issues in general for us and our customers
>
> Such as?

mostly a concern if we have to maintain other brances of boost than the 
library maintainers on boost.org.

>
> > - platform support in Boost ( or if the platform support Boost if you
> > like ;-)
>
> Sorry; what do you mean?

Just trying to be funny, sorry...  the pun was that if boost is standard 
conformant, then the platforms are to blame.   However, there are often some 
truth in what I try to joke about. 

>
> > All of these concerns can be declared acceptable, given that you are
> > willing to accept trade offs and risk.  The issue I try to raze is if
> > it is possible to reduce the magnitude of these risks and trade offs.
>
> Good question.
>
> > One of the key issues is that you have to decide on some strategy of
> > how to maintain your boost version.  The most obvious strategy may be
> > to stay with the latest Boost release at any time, this will however
> > add risk as you have to adopt new changes as they are introduced into
> > Boost and not as it fits your own release plans.  The obvious
> > alternative is to stay with a Boost release longer, possibly until you
> > are forced to switch. This strategy will quickly force you into
> > maintaining your own branch as Boost developers can not be expected to
> > support old releases.
>
> A company offering commercial Boost support can support old releases
> by integrating fixes into older release branches.  Whether or not
> those branches can ever become "official Boost releases" is not clear,
> but even if they don't, at least you can share a common set of fixes
> with everyone else who is depending on that old release.
>

agree

> > Between these there are many shades of gray which may fit more or
> > less well depending on your project and your customers.  If your
> > customers themselves use boost you are likely to meet a added
> > configuration management, testing and release related cost and risk.
> >
> >
> > As library developers have a very different type of customer
> > relationship than what is the case for end application developers
> > there are a number of concerns that make adaptation of Boost, or any
> > other 3rd party library for that sake, more problematic for library
> > developers.  The main issue is to what degree a library depends on
> > other libraries and hence introduces indirect or direct dependencies
> > to its users.  These dependencies tend to be troublesome for the
> > library developer to support, as well as they may be troublesome for
> > users to accept.  In fact users may easily be in situations where such
> > dependencies are completely unacceptable.  To help to understand the
> > issue I like to define categories of libraries which affect the degree
> > they are affected by these concerns.
> >
> >
> >
> > Cat1. open source, source distributions, allowing encapsulation
> >
> > Cat2. open source, source distributions in general
> >
> > Cat3. open source, binary distributions
> >
> > Cat4. closed source
> >
> >
> >
> > Cat1 and Cat2 are build-able source code distributions.
>
> I believe there are closed source libraries that are also delivered
> int the form of buildable source... but that's probably beside the
> point.

Depend of the definition of closed source, I consider it a Cat3 if it 
technically is buildable, but you have no licence to modify and then build.  
Often source code is open for debugging purposes only.

>
> > This give the user the ability to build the library and ensure that
> > it is built correctly with other library it depends on, on the
> > target system.  Cat1 add to this the ability to hide the use of it
> > inside other libraries, i.e. it can live concurrently in an end
> > system with other versions of the same Cat1 library.
>
> E.g. via symbol isolation or namespace renaming?

That is what we have in the toolbox, at least as far as I am aware of, yes.

>
> > Boost is a Cat2 library.  Even if the extensive use of headers only
> > libraries help some of the issues addressed in this post, it is not
> > a general solution that makes Boost or part of Boost a Cat1 library.
> >
> > Now an example showing some dependencies:
> >
> >
> >
> > MyCustomersApplication
> >
> >   |---- Boost 1.33
> >   |
> >   |---- MyLib
> >   |
> >           |---- Boost 1.32
> >
> > If MyLib is Cat1 or Cat2 the chances are good that we may be ok.  Most
> > likely MyLib will soon adopt the migration to new Boost releases by
> > the actions of the maintainer or by patches from end users.  If
> > however MyLib is Cat 3 or 4 we have a problem.  This problem is real
> > even if the use of Boost in MyLib is strictly internal.  In my view
> > this is an especially bad problem.  Even if I do not intend to use
> > Boost in any way in the API of MyLib I still have this issue of
> > incompatibility.
>
> Not if you maintain symbol isolation.  Of course, that may not be
> possible for you.

I do not know how on all platforms I have to deal with.

>
> > Ok, back to the example.  If MyLib have no intention of using Boost in
> > the API, is there a way?
>
> There are several ways to maintain symbol isolation.  On Windows, it's
> fairly easy: simply use a shared library and don't dlexport anything
> involving boost.  On *nix, recent GCCs can give you control over
> symbol visibility, so you can play the same kind of game.  Otherwise
> -- and this is inconvenient at best -- you can do it by using dlopen
> to load a shared library containing the boost symbols, and speak to
> that library through one or more known entry points.  That approach
> can be followed on Windows as well (using LoadLibrary) if symmetry
> with *nix turns out to be more appropriate than using the platform
> facilities.

Use of dlopen/dlsym and the win32 equallents is sertainly possible.  Good 
point -- I will think more about that one.  This could be appropriate if 
internal use of boost can be statically linked into a loadable library, and 
that module's services can be accessed through simple methods, alternatively 
through some framework - component hook-up.  Interfaces to the module have to 
be Boost less.

>
> > Side Note:
> >
> > I am aware of symbol encapsulation abilities in windows DLLs and
> > similar new features supported by GCC, however these are not generally
> > supported on all platforms, so I have selected to ignore them. They
> > may however be helpful to many.
>
> Okay, then you're stuck with the "plug-in" approach using dlopen.

yes, and renaming namespaces..

>
> > The general idea is that there are no references to symbols in the
> > boost namespace between any MyLib libraries, nor in any header files
> > in or referenced from MyLib API.  How can this be achieved?  I have
> > concluded that accessing boost trough namespace wrapping header files
> > are unpractical at best.  This needs to be designed into all of boost
> > for this to work, and I do not know exactly how.  The main challenge
> > is #include <boost/xxxx.hpp> directives which does not refer to the
> > wrapping headers.
>
> Other libraries that allow that sort of thing have all of their
> headers #include both a prefix AND a suffix header.  It would be a big
> change.

Yes, and I do not like the prospect of that.

David, you left out my text about use of bcp to rename the boost namespace 
here, that is what I wrap up.

> > So to wrap up:
> >
> > I do not believe anything prevents me from going along this path if I
> > decide to do so.  I am however interested in the boost developers
> > attitude toward this approach, and in particular if this is something
> > you will encourage and/or support.  I am willing to work on the tool
> > support for this in boost or to maintain a modified bcp tool if that
> > is more appropriate.
>
> I have mixed feelings about this.  One of the things some people
> complain about when they look at the boost code is that the many
> workarounds get in the way of understanding.  In fact, maintainability
> was one of the issues you listed at the beginning.  So if we put in
> facilities for namespace renaming, our code would get uglier and less
> transparent.  Maybe that's not a big deal, as we've already stepped
> off the "workaround cliff," but it is a concern for me.

Ok,  I agree with everything you write above, except the part about somebody 
falling off a cliff.  But I feel like you missed one important point I tried 
to make.  What I try to propose is something requiering no change in the way 
boost source code is written or maintained.  I propose tool support for a 
_production_step_ ensuring namespace renaming in a subset of boost selected 
with the bcp tool.  I would strongly advice all users of this tool support 
_not_ to maintain boost sources in the form where namespaces are renamed.  I 
think of the original boost code as a model for generation of buildable 
source code.  Kind of like CORBA IDL for idl2cpp compilers.  Nowbody with 
their sences intact would modify the generated C++ binding code, at least not 
manually. 

--
Bjørn