Re: [Boost-users] tweaking the review process (was: signals2 review results)

Subject: Re: [Boost-users] tweaking the review process (was: signals2 review results)
From: John Phillips (phillips_at_[hidden])
Date: 2008-11-21 12:50:16

   There are a number of things in here I want to reply to. However, I 
want to preface my reply by saying these are my personal opinions, they 
are not statements of policy from a Review Wizard.

                        John

Stjepan Rajko wrote:
> On Thu, Nov 20, 2008 at 11:40 PM, vicente.botet
> <vicente.botet_at_[hidden]> wrote:
>> I would like to make some suggestion to improve the review management:
>>
> 
> Thank you for starting this discussion.

   Thanks to both of you for starting it and bringing it out where more 
people will notice.

> 
>> * I had notice only at the end of the the review that the review tokes place on two mailing lists (devel ans user). Maybe this is usaual on Boost reviews but I was not aware. It will be more transaparent if all the reviews were posted to the same mailing list, maybe a specific one should be created.
> 
> [snip]

   Yes, it does. The problems Stjepan points out with a single location 
are largely the motivation for the multi-list discussion. Realistically, 
there are people who only have time or interest in one of the two lists, 
and will not sign up for both. Even worse would be a list that only 
exists for reviews, that would quickly become a ghost town.

   There are multiple populations of interest in Boost. There are those 
who are interested in hashing out all of the development details and 
those who are more interested in using the libraries without needing to 
discuss the details. Both have important perspectives to offer to a 
review, and both should be encouraged to participate. One way to do this 
is to keep the barrier for such participation as low as is reasonably 
possible. It means that the library author and the review manager are 
committing to watching both lists (as well as anyone who wants to see 
all parts of the conversation), but that is putting the extra effort on 
the people most dedicated to doing it and is a better choice.

> 
>> * From the 5 committing reviewers making this review possible only 3 made a review and 2 of them late. I'm wondering if this new rule must be preserved as the review can be accepted without the commiting reviewers review.
> 
>[snip]
 >
>
> So, I feel that committed reviews are a good way of reasonably making
> sure that a certain number of reviews will be submitted.  I'm not sure
> that the library acceptance / rejection should be influenced by the
> number of committed reviews (reviews from others are a perfectly good
> substitute).
> 
>[snip]

I feel pretty strongly that the committed reviewers not getting a chance 
to submit should not disqualify the library. Nor should the reviews 
coming in late.

> Another thing... many of the committed reviewers reported that they
> were first time reviewers (and the reviews they submitted were
> impressively detailed and very valuable). Personally, I was thrilled
> by this, since getting new participation in boost is critical to
> sustaining its quality .  Perhaps some combination of allowing
> reviewers to commit beforehand, and explicitly encouraging reviews
> focused on the user perspective (another suggestion from the
> http://tinyurl.com/48tdjs thread), helped in getting first-time
> reviewers (perhaps the reviewers can comment on this?).

   I'm happy about it, as well.

> 
>> * There were some reviews that came into this list by the intermediation of the review manager with a delay between the posting of the reviwer and forward from the RM. One negative review posted the 4th and reveived in this list the 11th other positive posted the 2nd and received in this list 3rd. I think that the review manager should not encourage the reviewers to send the reviews to himself. This will avoid this kind of delays. So I purpose that only the reviews sent to this single mailing list must be taken in account.
> 
> Yes, this particular delay is entirely my fault.  I can't explain why
> I didn't notice this email until so late (which is the point where I
> contacted the poster notifying him I would like to forward the mail,
> and forwarded it the following morning).
> 
> Allowing reviews to be sent to the review manager is straight from
> http://www.boost.org/community/reviews.html (Introduction paragraph).
> There are two valid reasons for this, IMO:
> 
> * the reviewer is only subscribed to one of {boost, boost-user,
> boost-announce} lists, and would like the reviewer to be forwarded to
> both boost and boost-user lists (if there was a dedicated review list,
> like you suggest, that was also open to all posters, then this reason
> would go away)

   Sometimes reviews even come from people who aren't subscribed to any 
of the lists (This is not common, but it has happened.) If they have a 
thoughtful and productive review to offer, they should be encouraged, 
not excluded.

> 
> * the reviewer would like to remain anonymous to the list (granted,
> this could also be accomplished by sending from an anonymous e-mail).
> 
> If this stays as it is, the RM should be more diligent about
> monitoring her personal mail, which I apparently wasn't.
> 
> 
>> * Even if the review was over the 10th there were 2 accepting reviews comming from the commiting reviewers just some hours before the review result annoncement 19th. I think that the review manager must state clearly when the review is over and do not accept any review after. This do not means that the RMcannot change this date, but annnce it clearly.
> 
> I tried to keep my timeline as transparent as possible.  In my
> review-closing email, I asked people to let me know if they were
> considering writing a review.  Those that contacted me were informed
> of the timeline of pending reviews.  When the last promised review was
> submitted, I set a hard deadline.  Given the uncertainty in people's
> schedules, I decided to approach this in a flexible way.  That was
> just my personal preference - I'm sure setting a hard deadline earlier
> would have been a fine choice as well.

   Let's imagine for a moment what a hard cutoff with everything after 
that ignored would mean. What if, the day after the review period 
closed, someone submitted a late review that showed conclusively that 
the library was not usable? (Maybe it has unacceptable side effects that 
no one else noticed, or maybe it infringes on a patent that none of the 
rest of us knew about. Whatever reason, it can't be allowed to pass.)

   Even if every review was positive and suggested acceptance during the 
review period, this should fail. Drawing a line and saying "nothing 
after this date" would make this scenario possible.

   In fact, asking for such a hard cutoff shows one misunderstanding 
about the current process. As the process currently exists, it is not 
democratic. It is a benign and hopefully well informed autocracy. The 
Review Manager has the authority to over rule the plurality of 
reviewers. The goal of the reviews is not to collect votes, but instead 
to provide the best possible information for the manager. We select 
managers with the intention that they will make good decisions given 
this information.

   (I know of no circumstance where a manager has been overruled by the 
Wizards or the Moderators, but I do follow the discussion on every 
review to make sure that I think the decision reached by the manager is 
reasonable. If I found something that lead me to believe the decision 
was unreasonable, I would start a discussion with the other Wizard and 
the Moderators about what to do. On the one occasion where a manager did 
not fulfill the obligation to decide, I had to step in and finish the 
review.)

   Since the goal of the review process is to inform the manager as well 
as possible, cutting off the discussion arbitrarily is counter 
productive. In fact, in the reviews I have managed I have intentionally 
looked at posts from both before and after the official review period 
for extra information. The only cutoff that can't be avoided is that I 
try to submit the review results within about a week or so of the end of 
the period. That obviously cuts off what I can consider.

> 
>> I hope this will help to improve future reviews,
>>
> 
> Thanks again for starting this discussion and for your suggestions - I
> also hope good improvements will come out of it.
> 
> Stjepan

   Thanks to both of you. Considering how to do the reviews better is 
always important.

                        John