Re: [boost] RE process (prospective from a retired FreeBSD committer)...

Subject: Re: [boost] RE process (prospective from a retired FreeBSD committer)...
From: Dave Abrahams (dave_at_[hidden])
Date: 2011-01-30 17:16:37

At Sun, 30 Jan 2011 18:06:28 +0300,
Vladimir Prus wrote:
> 
> Dave Abrahams wrote:
> 
> > At Sun, 30 Jan 2011 16:43:17 +0300,
> > Vladimir Prus wrote:
> >> 
> >> Maybe, stricter published guidelines, together with automatic
> >> integration testing and nagging and reverting, is what we need here.
> >> 
> >> > Secondly, I don't see having "a single trunk" with these
> >> > properties as bringing with it any particular "goodness."  If
> >> > you're looking for a fix you still need to know which code to look
> >> > at.  So just go to that library's GitHub repo.
> >> 
> >> What if you don't know which library is that?
> > 
> > A big monolithic repo has the same problem.
> 
> Not really -- you need a single checkout operation to find whether
> the problem is fixed or not with 99% probability. And only then, to
> investigate further.

You mean, via a "try it and see" approach?

Making it a single user operation to update to the latest released
version of any or all libraries is under my proposal so trivial as to
be unworthy of much discussion IMO.

> >> > But in reality, it doesn't tell you anything about the next
> >> > release of Boost, because things don't always get moved from trunk
> >> > to release.
> >> 
> >> That's the consequence of our non-standard release process. All
> >> other software projects create release branches off trunk, and Boost
> >> is the only one I know who has long-lived release branch. I though
> >> it was a conscious decision, but all the recent discussions makes me
> >> doubt that, so maybe we should briefly discuss relative merits.
> >> 
> >> 1. The 'make release branch off trunk' process is good for all
> >> purposes, really, except it requires that at some point in time,
> >> trunk is in good shape. Historically, for lack of tools and release
> >> management resources, we often could not get trunk in good shape
> >> quick.
> > 
> > And that requires far too much coordination to be practical on a
> > project of this size.
> 
> I don't think this statement has been proven.

Nothing in this discussion is going to be "proven."

> >> 2. Ours 'merge to release branch' process has the advantage that
> >> even if somebody breaks library X in trunk completely, he might
> >> just forget to merge it to release. So, we get improved stability,
> >> at the cost of slow progress. It's often the case that somebody
> >> forgets to merge changes to release branch, especially for
> >> patches and fixes applied outside of one's official libraries.
> >> 
> >> So, in comparison, to achieve the same rate of changes and quality,
> >> (1) requires discipline with commits and checking test results,
> >> while (2) requires the same, and additional dancing and coordination
> >> with merges. So, (2) is strictly more complex and error prone, and
> >> should only be done if specific maintainers want separate branches
> >> for their components.
> > 
> > Anyone can make a separate branch at any time no matter which
> > procedure is used.
> 
> And? (2) still requires more effort.

No argument.  I am just saying that the one condition under which you
say 2 "should only be done" isn't even really valid.

> >> >> What you propose breaks this useful thing, because:
> >> >> 
> >> >> - Only release managers "pull" into the unified thing
> >> > 
> >> > Not really.  The individual library developers determined which
> >> > versions will be automatically pulled into the unified thing by
> >> > tagging their individual releases as STABLE.
> >> 
> >> There are two problems here:
> >> 
> >> 1. Just like library developers forget to merge to release, they
> >> will forget to tag those 'releases'
> > 
> > That's totally up to the library developer.  Library developers
> > generally care Boost contains an up-to-date version of their
> > libraries, so they will be motivated to remember.
> 
> "will"? In practice, right now, developers do forget. 

This line of argument eventually devolves into "library maintainers
will forget to check their changes in at all, so they should work on a
network share that we snapshot for release."  I think marking a change
as ready for release should be a fairly conscious decision, and one
that's separate from saying "let's test this code out."  As long as
that's the case, someone can forget.  IMO the trade-off is well worth
it.

> And again, there are often commits made by those who are not
> official maintaners of library X (because X might not even have
> maintainers). The chances of such changes of falling through are
> even higher.

I don't see how.  Right now someone has to explicitly decide which
changes are moved to the release branch, and something equivalent
will happen 

> >> 2. Because 'git cherry-pick' is fundamentally unable to record
> >> any mergeinfo, this means that any time you want to merge a
> >> single specific fix to release branch, you would have problems.
> > 
> > What release branch?  Who said anything about cherry-pick?  What
> > problems?
> 
> Suppose you have library X with 200 new changes. For next release, it is
> necessary to include one of those changes. 

You cherry-pick that commit onto the branch in X's repo that currently
points at X's last release.

> How do you do that? With current release process, it's a single
> command (obviously, run by a single person).
>  
> > And regardless, SVN has all the same issues w.r.t. picking
> > individual changes, doesn't it?
> 
> No. When you merge a single revision from branch A to branch B, SVN
> records accurate information about this merge.  When you do 'git
> cherry-pick', git does not record any information whatsoever and is
> fundementally unable to record it.

Ah.  I was just complaining to a friend about the fact that Git drops
that information.  Mostly I don't miss it for any practical purpose,
but conceptually I'd like it to be there.

> >> >> - Release managers only pull "releases",
> >> > 
> >> > Which can be much more frequent than what we do at Boost.
> >> 
> >> Oh. I think everybody thinks that our current rate of releases is
> >> almost-too-often already.
> >
> > Surely you realize that the release rate for
> > individual libraries does not have to affect the release rate for
> > Boost?
> 
> First, in "release managers only pull releases", above, we are
> surely talking about Boost release managers?

Yes.  We are talking about them "pulling" the releases of individual
libraries, not of Boost.

> Second, could you name, exactly, all libraries whose developers
> expressed a need for individual releases? 

No, of course I can't name all of them exactly.  Can you?  I doubt it.

> I know of one.

I know I have expressed that need, and I've heard others express it.

> >> >> which, together with delays added by release managers, means that
> >> >> a fix to a component will be added to the unified thing after
> >> >> considerable delay
> >> > 
> >> > I don't see where the delay comes from.
> >> 
> >> Either release managers look at the thing they merge (which adds
> >> considerable delay), or they don't, in which case direct push to
> >> release branch is more efficient.
> > 
> > They don't look, unless there are test failures or other problems.
> > And they don't merge.  At most we're talking about updating submodule
> > references.
> 
> Which still has to be done manually?

No.  From above: "The individual library developers determined which
versions will be automatically pulled into the unified thing" by which
I mean there's a "button" release managers can push to assemble a
collection of the latest release versions of all the libraries in the
collection.

> >> Strangely, I know of other examples. Say, KDE, which is zillion
> >> times bigger than Boost, has relatively small number of separate git
> >> repositories,
> > 
> > They're trying to fix that, IIUC.
> 
> Oh? It does not seem like any split of kdelibs is immediately
> forthcoming, or even discussed with such activity as we're having
> here. 

IIUC there is strong interest within at least part of the KDE
community in following a Ryppl-like modularized approach.  They have
been looking into various kinds of package management infrastructure
for this job.  I'll be happy to try to find out more if you want.

> Maybe, that suggest that we're actually solving wrong problem?

Exactly the opposite.  KDE's moving in that direction IIUC; FreeBSD is
moving in that direction.  I'm confident we could quickly find 10
other projects that are trying to modularize and zero modularized
projects that are trying to switch to a monolithic organization.  When
have you ever heard of that happening?

> > The problem isn't where the code is located, it's:
> > 
> > a. how much coordination is required to get things done
> 
> Yes. And given that right now, coordination basically boils down to
> library authors merging their things before deadline (or not merging),
> and given that merge is a single command, 

svn switch, svn merge, svn switch == a single command?  But anyway,
that's irrelevant because...

> it's not clear to me how you are improving on that.

...it's not about the number of commands, it's about the sense of
speed and freedom with which library developers can operate.

> > b. the mental drag of having to deal with all the parts of Boost
> >    that simply aren't relevant to you.
> 
> What do you mean by mental drag? Having extra directories in your
> checkout?

Yes, among other things.  For me personally, Boost's monolithic
structure takes much of the fun out of working on it.  Totally
subjective and un-provable, I know.

so-sue-me-ly y'rs,

-- 
Dave Abrahams
BoostPro Computing
http://www.boostpro.com