Re: [boost] boost organization and installation

From: William E. Kempf (williamkempf_at_[hidden])
Date: 2002-05-02 16:25:28

----- Original Message -----
From: "Noel Yap" <yap_noel_at_[hidden]>
To: <boost_at_[hidden]>
Sent: Thursday, May 02, 2002 3:37 PM
Subject: Re: [boost] boost organization and installation

> --- "William E. Kempf" <williamkempf_at_[hidden]>
> wrote:
> > > At least we're on the same page on this one --
> > this is
> > > a process problem.  It's great you're working on
> > > solving it.
> >
> > Yes, but it's not an easy problem to solve.  AFAIK,
> > Boost is the first
> > project of its type.  It has specific needs and
> > requirements that don't
> > exist for other projects.  As such, though we can
> > and should pay attention
> > to successful processes used by other projects, none
> > of them quite fit what
> > we need.  So, we're limping along a bit here as we
> > try and define a process
> > that will work for us.  We'd gladly welcome help
> > from anyone who wants to
> > volunteer... but before you volunteer I'd suggest
> > learning what our specific
> > needs are by wading through the archives, static to
> > noise ratio problems or
> > not.
>
> I really think such requirements need to be
> documented, especially for this project.

Granted.  Maybe you can document them as you wade through the archives?  We
can update them after that.

> > > I think once the release process is improved, it
> > will
> > > diminish, but not eliminate, my stability concerns
> > --
> > > there's still many pieces that doesn't concern me
> > that
> > > could introduce instability.
> >
> > You can never eliminate them, unfortunately, because
> > of the unique nature of
> > Boost.  We should be able to get things to be much
> > better then they are now,
> > however.  For instance, the problems with
> > Boost.Threads occurred solely
> > because it was a new library and I had few other
> > people to rely on testing
> > things on other platforms for me.  A compiler farm
> > or automated build
> > testing process would have helped here, and there's
> > an effort to bring this
> > to reality as well, but that's a non-trivial thing
> > to do for a project this
> > large that exists solely as a volunteer effort with
> > no monetary backing at
> > all.
>
> This is my point.  I downloaded 1.27.0 which had a
> broken Threads component.  As a CM I cannot bless this
> release for our use even though the other portion that
> we actually do use may be fine.  If I could get a
> minimal package that I can install, I wouldn't have a
> concern whether another portion of Boost built or not.

I'm not a CM, but this stance seems somehow wrong.  Although to address this
there'd have to be a way to install only the portions you need, in addition
to building only the portions you need.  That's an area where we're lacking,
currently.

> > > Why do you say that?  If Boost depends on
> > > Boost.Python, and Boost.Python depends on Python,
> > then
> > > Boost depends on Python (ie dependency is
> > transitive).
> > >  Would you agree that dependency is transitive?
> >
> > There's no such library as "Boost", which may be why
> > you don't understand
> > what we're saying here.  Boost is a collection of
> > libraries, much like the
> > Apache/Jakarta libraries you mentioned as models for
> > a split.
>
> Yes, exactly.  I think each independent piece of Boost
> should be installable separately.

That's a worthy goal, but it can't impact the Boost development efforts.
That's why I'm proposing a seperate effort to address distribution needs.

> >  I don't think
> > that model will work for us, because of our unique
> > goals and requirements,
> > but despite that there is no "Boost library".
>
> Of course, I can't comment on this since I know
> nothing of the requirements, but right now, I don't
> see why it can't be treated completely as separate
> components (it seems that it's already treated like
> this partially), in which there are core components
> (ie config) and the other components which may depend
> on some other components.  Each component can be
> tarballed into their own dist package allowing
> separate installations.  One giant tarball can also be
> created for those not interested in small granularity.

Theoretically this could be accomplished.  The problem is that the
dependencies sometimes are either internal or external (i.e. they're needed
by either the interface or the implementation) which can impact what's
required for distribution.  More over, the dependencies can change on a whim
(especially when they are internal dependencies).  Considering the goals of
Boost I don't think we can expect any more out of the developers then
documenting and/or e-mail dependency changes to someone.  We can't expect
them to change any distribution procedures/scripts that occur because of
these changes.  (That is, other then changing any Jamfiles if required by
the library.)

> >  Boost.Threads
> > and Boost.RegEx (the only
> > other libraries that are currently built, AFAIK)
> > will function just fine
> > with out Boost.Python.  In fact, you can choose to
> > build the seperate
> > libraries instead of trying to build the entire
> > Boost project simply by
> > running Jam in the specific library's build
> > directory.  Maybe this solution
> > will work better for you in the sort term?
>
> Yes, this will cover half the problem.  The other half
> is installation.  I think the installation of the
> actual libraries is simple, it's the header files I'm
> more concerned about.  All the header files are
> packaged in one directory.  Aside from
> http://www.boost.org/libs/hdr_depend.html, is there
> documentation on which header files belong to which
> projects?

No, and in some cases this is a fluid thing (i.e. the dependencies change
intermittently, or even frequently).

> > Because management of downloads for individual
> > libraries is something that
> > we can't handle... partially because there are
> > interdependencies.
>
> I see.  I think we differ on this issue.  I think it
> is the developer's responsibility to document
> dependencies and the user's responsibility to install
> any dependencies (JDE is distributed this way).  Any
> circular dependencies should be contained within the
> same package, but from what I see, there are no
> circular dependencies among Boost components.

The difference in view points is that as a developer I deal with Boost as a
large package (though I don't care if certain parts fail to build... heck I
often don't build the parts I don't care about).  I'm not concerned with
(nor should I be, because my end goal isn't distribution/use) how to manage
dependency information required to install portions of Boost.  From my
perspective what's needed is either a seperate effort that deals with these
issues with out impacting me, or an automated system for dealing with these
issues that minimally impacts both my time and the requirements on tools
that need be present on my system.  The Boost.Build system may be able to
address some of the automated system requirements that distributions will
need, but where it can't I think we'll need the seperate volunteer effort.

> > > Why complicate the entire Boost build due to a
> > > dependence of only one of its components?  Will
> > this
> > > scale if more components have more dependencies?
> > Will
> > > it scale as well as if each library took care of
> > their
> > > own dependencies?
> >
> > And how would you propose they do that?  Maybe there
> > is a solution that
> > would allow for this sort of distribution... but you
> > have to keep in mind
> > some facts about Boost.  You can't complicate the
> > lives of the developers to
> > the point that they no longer contribute because
> > they don't have the time.
>
> IMHO, it shouldn't complicate anything at all and
> might even simplify things.  All that's really needed
> (although new namespaces would be nice), is some a
> tarball script that would package up the source and
> documentation.  I think the only real change might be
> a reorg of the code (although I would imagine that
> each project might already have separate source trees
> for their stuff).

A reorg of the code could well be something that we can't do.  Depends on
how extensive the reorg is and whether such a reorg will make development
efforts more difficult.  Currently the structure is optimized for
maintainance by developers.

I also forsee other problems that will complicate the development process.
If the "tarball script" has to be maintained by the developers as libraries
change, then that's exactly the kind of impact that I believe may well be
unacceptable.  If the changes were trivial and easily done then it might not
be an issue, but the solutions proposed so far for distribution don't fit
this description.  That's why I believe there has to be a seperate effort by
other volunteers.

> > We want our libraries to be used, because this helps
> > us evolve the libraries
> > over time, but that's not our end goal.  Our end
> > goal is to extend the
> > language and it's libraries.
>
> Ahhh.  This is the heart of the matter.  I've been
> working under the assumption that usage was the end
> goal.

Yes, that truly is the heart of the matter.  Usage is great for us, and we'd
love to increase it.  But it's not the end goal and if the developers have
to put too much time into making Boost more accessible we may very well find
some of them leaving.

> > I'd encourage such an effort... but I'd caution you
> > to pay attention to the
> > unique needs of Boost in this regard.  So, again,
> > I'd suggest you wade
> > through the past threads on this subject.  I'll go a
> > step further... I'm
> > willing to help any group of users that want to
> > start a BoostUsers
> > organization to handle distribution issues.  I can
> > help you to understand
> > our needs (provided you do some of the legwork
> > yourself... my time is
> > limited), help to coordinate with the Boost
> > membership, etc.  I think this
> > would be a nice addition to the current BoostUsers
> > mail list, which I'm a
> > moderator of any way.  I think everyone could
> > benefit from such an effort.
>
> This would be great.  But, yes, I really do need to
> understand the requirements more.  The fact that the
> end goal is to put something in the standard makes me
> want to reevaluate both the need and the want to
> separate out the packages.

I believe that in the end a reevaluation may well result in the same desire.
However, we need to find a way to do it that won't impact the actual goals
of Boost.

> My first thought on this is that separation would make
> it easier for the standards body to evaluate each
> package separately.

For the most part they evaluate a library's design, not it's implementation.
The libraries have been designed in a manner that makes it possible for
evaluation according to this criteria.  Having seperate installation
packages for the various libraries may or may not help in such an
evaluation, but the current state is good enough.

> I'll start digging through the arhives ASAP (although,
> like you, I'm busy with other stuff as well).

Quite understandable... but that's part of the reason why there isn't a
solution for distribution yet. :(

Bill Kempf