Boost logo

Boost Users :

Subject: Re: [Boost-users] tweaking the review process (was: signals2 review results)
From: John Phillips (phillips_at_[hidden])
Date: 2008-11-21 12:50:16

   There are a number of things in here I want to reply to. However, I
want to preface my reply by saying these are my personal opinions, they
are not statements of policy from a Review Wizard.


Stjepan Rajko wrote:
> On Thu, Nov 20, 2008 at 11:40 PM, vicente.botet
> <vicente.botet_at_[hidden]> wrote:
>> I would like to make some suggestion to improve the review management:
> Thank you for starting this discussion.

   Thanks to both of you for starting it and bringing it out where more
people will notice.

>> * I had notice only at the end of the the review that the review tokes place on two mailing lists (devel ans user). Maybe this is usaual on Boost reviews but I was not aware. It will be more transaparent if all the reviews were posted to the same mailing list, maybe a specific one should be created.
> [snip]

   Yes, it does. The problems Stjepan points out with a single location
are largely the motivation for the multi-list discussion. Realistically,
there are people who only have time or interest in one of the two lists,
and will not sign up for both. Even worse would be a list that only
exists for reviews, that would quickly become a ghost town.

   There are multiple populations of interest in Boost. There are those
who are interested in hashing out all of the development details and
those who are more interested in using the libraries without needing to
discuss the details. Both have important perspectives to offer to a
review, and both should be encouraged to participate. One way to do this
is to keep the barrier for such participation as low as is reasonably
possible. It means that the library author and the review manager are
committing to watching both lists (as well as anyone who wants to see
all parts of the conversation), but that is putting the extra effort on
the people most dedicated to doing it and is a better choice.

>> * From the 5 committing reviewers making this review possible only 3 made a review and 2 of them late. I'm wondering if this new rule must be preserved as the review can be accepted without the commiting reviewers review.
> So, I feel that committed reviews are a good way of reasonably making
> sure that a certain number of reviews will be submitted. I'm not sure
> that the library acceptance / rejection should be influenced by the
> number of committed reviews (reviews from others are a perfectly good
> substitute).

I feel pretty strongly that the committed reviewers not getting a chance
to submit should not disqualify the library. Nor should the reviews
coming in late.

> Another thing... many of the committed reviewers reported that they
> were first time reviewers (and the reviews they submitted were
> impressively detailed and very valuable). Personally, I was thrilled
> by this, since getting new participation in boost is critical to
> sustaining its quality . Perhaps some combination of allowing
> reviewers to commit beforehand, and explicitly encouraging reviews
> focused on the user perspective (another suggestion from the
> thread), helped in getting first-time
> reviewers (perhaps the reviewers can comment on this?).

   I'm happy about it, as well.

>> * There were some reviews that came into this list by the intermediation of the review manager with a delay between the posting of the reviwer and forward from the RM. One negative review posted the 4th and reveived in this list the 11th other positive posted the 2nd and received in this list 3rd. I think that the review manager should not encourage the reviewers to send the reviews to himself. This will avoid this kind of delays. So I purpose that only the reviews sent to this single mailing list must be taken in account.
> Yes, this particular delay is entirely my fault. I can't explain why
> I didn't notice this email until so late (which is the point where I
> contacted the poster notifying him I would like to forward the mail,
> and forwarded it the following morning).
> Allowing reviews to be sent to the review manager is straight from
> (Introduction paragraph).
> There are two valid reasons for this, IMO:
> * the reviewer is only subscribed to one of {boost, boost-user,
> boost-announce} lists, and would like the reviewer to be forwarded to
> both boost and boost-user lists (if there was a dedicated review list,
> like you suggest, that was also open to all posters, then this reason
> would go away)

   Sometimes reviews even come from people who aren't subscribed to any
of the lists (This is not common, but it has happened.) If they have a
thoughtful and productive review to offer, they should be encouraged,
not excluded.

> * the reviewer would like to remain anonymous to the list (granted,
> this could also be accomplished by sending from an anonymous e-mail).
> If this stays as it is, the RM should be more diligent about
> monitoring her personal mail, which I apparently wasn't.
>> * Even if the review was over the 10th there were 2 accepting reviews comming from the commiting reviewers just some hours before the review result annoncement 19th. I think that the review manager must state clearly when the review is over and do not accept any review after. This do not means that the RMcannot change this date, but annnce it clearly.
> I tried to keep my timeline as transparent as possible. In my
> review-closing email, I asked people to let me know if they were
> considering writing a review. Those that contacted me were informed
> of the timeline of pending reviews. When the last promised review was
> submitted, I set a hard deadline. Given the uncertainty in people's
> schedules, I decided to approach this in a flexible way. That was
> just my personal preference - I'm sure setting a hard deadline earlier
> would have been a fine choice as well.

   Let's imagine for a moment what a hard cutoff with everything after
that ignored would mean. What if, the day after the review period
closed, someone submitted a late review that showed conclusively that
the library was not usable? (Maybe it has unacceptable side effects that
no one else noticed, or maybe it infringes on a patent that none of the
rest of us knew about. Whatever reason, it can't be allowed to pass.)

   Even if every review was positive and suggested acceptance during the
review period, this should fail. Drawing a line and saying "nothing
after this date" would make this scenario possible.

   In fact, asking for such a hard cutoff shows one misunderstanding
about the current process. As the process currently exists, it is not
democratic. It is a benign and hopefully well informed autocracy. The
Review Manager has the authority to over rule the plurality of
reviewers. The goal of the reviews is not to collect votes, but instead
to provide the best possible information for the manager. We select
managers with the intention that they will make good decisions given
this information.

   (I know of no circumstance where a manager has been overruled by the
Wizards or the Moderators, but I do follow the discussion on every
review to make sure that I think the decision reached by the manager is
reasonable. If I found something that lead me to believe the decision
was unreasonable, I would start a discussion with the other Wizard and
the Moderators about what to do. On the one occasion where a manager did
not fulfill the obligation to decide, I had to step in and finish the

   Since the goal of the review process is to inform the manager as well
as possible, cutting off the discussion arbitrarily is counter
productive. In fact, in the reviews I have managed I have intentionally
looked at posts from both before and after the official review period
for extra information. The only cutoff that can't be avoided is that I
try to submit the review results within about a week or so of the end of
the period. That obviously cuts off what I can consider.

>> I hope this will help to improve future reviews,
> Thanks again for starting this discussion and for your suggestions - I
> also hope good improvements will come out of it.
> Stjepan

   Thanks to both of you. Considering how to do the reviews better is
always important.


Boost-users list run by williamkempf at, kalb at, bjorn.karlsson at, gregod at, wekempf at