Re: [boost] RE process (prospective from a retired FreeBSD committer)...

Subject: Re: [boost] RE process (prospective from a retired FreeBSD committer)...
From: Vladimir Prus (vladimir_at_[hidden])
Date: 2011-01-30 10:06:28

Dave Abrahams wrote:

> At Sun, 30 Jan 2011 16:43:17 +0300,
> Vladimir Prus wrote:
>> 
>> Dave Abrahams wrote:
>> 
>> > First of all, I don't think there's general agreement that trunk
>> > should hold code "supposedly not worse in test results."  That's one
>> > reason our trunk test results are not all that useful.  Some people
>> > like to use trunk for "TDD," which apparently means checking in tests
>> > that are known to be broken and fixing them incrementally.
>> 
>> That seems like violating basic principles of software development.
> 
> It depends whose "basic principles" you subscribe to, apparently.

Of course -- it could be there are projects were you can commit broken
code to trunk any time you like, and everybody is fine with that -- and
it's just me who don't have enough experience to have met such projects.
(FAOD, "trunk" is not specific to any version control system, this is just
code that everybody is working on together to get shipped).

>> Maybe, stricter published guidelines, together with automatic
>> integration testing and nagging and reverting, is what we need here.
>> 
>> > Secondly, I don't see having "a single trunk" with these
>> > properties as bringing with it any particular "goodness."  If
>> > you're looking for a fix you still need to know which code to look
>> > at.  So just go to that library's GitHub repo.
>> 
>> What if you don't know which library is that?
> 
> A big monolithic repo has the same problem.

Not really -- you need a single checkout operation to find whether the
problem is fixed or not with 99% probability. And only then, to investigate
further.

>> >> Therefore, if I wish to make sure my code works with new release
>> >> of Boost, of if I wish to check if some bug is fixed, that is the
>> >> place to go.
>> > 
>> > But in reality, it doesn't tell you anything about the next
>> > release of Boost, because things don't always get moved from trunk
>> > to release.
>> 
>> That's the consequence of our non-standard release process. All
>> other software projects create release branches off trunk, and Boost
>> is the only one I know who has long-lived release branch. I though
>> it was a conscious decision, but all the recent discussions makes me
>> doubt that, so maybe we should briefly discuss relative merits.
>> 
>> 1. The 'make release branch off trunk' process is good for all
>> purposes, really, except it requires that at some point in time,
>> trunk is in good shape. Historically, for lack of tools and release
>> management resources, we often could not get trunk in good shape
>> quick.
> 
> And that requires far too much coordination to be practical on a project
> of this size.

I don't think this statement has been proven.

>> 2. Ours 'merge to release branch' process has the advantage that
>> even if somebody breaks library X in trunk completely, he might
>> just forget to merge it to release. So, we get improved stability,
>> at the cost of slow progress. It's often the case that somebody
>> forgets to merge changes to release branch, especially for
>> patches and fixes applied outside of one's official libraries.
>> 
>> So, in comparison, to achieve the same rate of changes and quality,
>> (1) requires discipline with commits and checking test results,
>> while (2) requires the same, and additional dancing and coordination
>> with merges. So, (2) is strictly more complex and error prone, and
>> should only be done if specific maintainers want separate branches
>> for their components.
> 
> Anyone can make a separate branch at any time no matter which
> procedure is used.

And? (2) still requires more effort.

>> >> What you propose breaks this useful thing, because:
>> >> 
>> >> - Only release managers "pull" into the unified thing
>> > 
>> > Not really.  The individual library developers determined which
>> > versions will be automatically pulled into the unified thing by
>> > tagging their individual releases as STABLE.
>> 
>> There are two problems here:
>> 
>> 1. Just like library developers forget to merge to release, they will
>> forget to tag those 'releases'
> 
> That's totally up to the library developer.  Library developers
> generally care Boost contains an up-to-date version of their
> libraries, so they will be motivated to remember.

"will"? In practice, right now, developers do forget. And again, there
are often commits made by those who are not official maintaners of
library X (because X might not even have maintainers). The chances
of such changes of falling through are even higher.

 
>> 2. Because 'git cherry-pick' is fundamentally unable to record any
>> mergeinfo, this means that any time you want to merge a single
>> specific fix to release branch, you would have problems.
> 
> What release branch?  Who said anything about cherry-pick?  What
> problems?

Suppose you have library X with 200 new changes. For next release, it is
necessary to include one of those changes. How do you do that? With
current release process, it's a single command (obviously, run by a
single person).
 
> And regardless, SVN has all the same issues w.r.t. picking individual
> changes, doesn't it?

No. When you merge a single revision from branch A to branch B, SVN
records accurate information about this merge. When you do 'git cherry-pick',
git does not record any information whatsoever and is fundementally unable
to record it.

>> >> - Release managers only pull "releases",
>> > 
>> > Which can be much more frequent than what we do at Boost.
>> 
>> Oh. I think everybody thinks that our current rate of releases is
>> almost-too-often already.
> 
> Now it seems like you're just looking for a snappy refutation without
> really thinking.  

Of course, that's precisely what I am doing all the time.

> Surely you realize that the release rate for
> individual libraries does not have to affect the release rate for
> Boost?

First, in "release managers only pull releases", above, we are surely
talking about Boost release managers?

Second, could you name, exactly, all libraries whose developers expressed
a need for individual releases? I know of one.

>> >> which, together with delays added by release managers, means that
>> >> a fix to a component will be added to the unified thing after
>> >> considerable delay
>> > 
>> > I don't see where the delay comes from.
>> 
>> Either release managers look at the thing they merge (which adds
>> considerable delay), or they don't, in which case direct push to
>> release branch is more efficient.
> 
> They don't look, unless there are test failures or other problems.
> And they don't merge.  At most we're talking about updating submodule
> references.

Which still has to be done manually?

>> >> therefore the unified thing we have now will no longer exists, and
>> >> two weeks before the release we'll get a frankenstein creature that
>> >> might technically pass all tests (though I doubt that), but won't be
>> >> tested by any users in real projects.
>> > 
>> > Welcome to the real world.  That's how every OS distribution and every
>> > other collection of semi-independent modules works, and works well.
>> > No sane user of a real project is going to slow down his development
>> > by regularly testing against an unSTABLE Boost trunk, and that's
>> > increasingly true as Boost grows.
>> 
>> Strangely, I know of other examples. Say, KDE, which is zillion
>> times bigger than Boost, has relatively small number of separate git
>> repositories,
> 
> They're trying to fix that, IIUC.

Oh? It does not seem like any split of kdelibs is immediately forthcoming,
or even discussed with such activity as we're having here. Maybe, that
suggest that we're actually solving wrong problem?

>> whereas most of the core (which is half-zillion times bigger than
>> Boost), lives in two modules, which contain wildly different
>> functionality, are updated by multiple people, and then built and
>> used and tested by users all the time. So, this proves that Boost
>> has a lot of grows ahead before having all code in place place will
>> becomes, even in theory, a problem.
> 
> The problem isn't where the code is located, it's:
> 
> a. how much coordination is required to get things done

Yes. And given that right now, coordination basically boils down to
library authors merging their things before deadline (or not merging),
and given that merge is a single command, it's not clear to me how
you are improving on that.

> b. the mental drag of having to deal with all the parts of Boost that
>    simply aren't relevant to you.

What do you mean by mental drag? Having extra directories in your checkout?

- Volodya

Vladimir Prus
Mentor Graphics
[phone]