Re: [Boost-users] [boost] tweaking the review process

Subject: Re: [Boost-users] [boost] tweaking the review process
From: John Phillips (phillips_at_[hidden])
Date: 2008-12-15 22:42:24

David Abrahams wrote:
> on Fri Nov 21 2008, John Phillips <phillips-AT-mps.ohio-state.edu> wrote:
> 
>> Stjepan Rajko wrote:
>>> On Fri, Nov 21, 2008 at 10:52 AM, John Phillips
>>> <phillips_at_[hidden]> wrote:
>>>> Stjepan Rajko wrote:
>>>>> [snip]
>>>>  Do you think it would help if the Wizards stressed this need and requested
>>>> an affirmation of it as part of the lead in to the review?
>>>>
>>> Definitely.  I'm not sure whether stressing this requirement would
>>> turn off potential review managers because of the added effort, but I
>>> think it would make the overall review process better.
>>>
>>> Stjepan
>>
>>   I don't want to discourage anyone who will do a good job of managing a review, but
>> if we decide this is a necessary step in providing a good review then I'm not upset by
>> discouraging anyone who won't do it.
> 
> I'm not sure just what's being proposed here (so this might be
> off-base), but it's important not to make anything unnecessarily
> burdensome.  We need to rely on peoples' intelligence and sense of
> responsibility without making egregious rules.  For example, a review
> manager should not -- in general -- need to do a comprehensive review
> herself; she has enough on her plate already.  95% of the time it should
> be enough to make a judgement based on the other reviews: both their
> votes and their quality, depth, etc.  At the end of the review period,
> if she is truly left unsure whether the library should be accepted, a
> review manager might go through the process herself, but otherwise a
> comprehensive review from the manager should not be required.
> 
> I think it would be good if the review wizards could make it clear that
> they are available to provide guidance in case review managers have
> questions about how to do their jobs well.
> 

   I think what Stjepan started this idea from is the part of the Review 
Manager description on the web site where it says

The Review Manager:

     * Checks the submission to make sure it really is complete enough 
to warrant formal review. See the Boost Library Requirements and 
Guidelines. If necessary, work with the submitter to verify the code 
compiles and runs correctly on several compilers and platforms.

   My reply was based on believing that is what he meant, at least.

   This is not, in general the same thing as preparing a full review, 
but it is also not trivial requirement. When I manage a review I take it 
to mean that I should read through the documentation and make sure it is 
sensible, check a few files to make sure the required license statement 
is there (hoping that a small sample is representative), and try to 
compile at least the examples to make sure it works on some selection of 
the compilers I can use.

   Some managers have taken this very seriously, I think. In my time as 
a Wizard I have seen times when the manager postpones the review with a 
request to work with the author to improve the library before 
proceeding. There have also been a couple of cases where the review 
comments have indicated that the reviewers did not think the library was 
ready for a full review.

   My understanding of Stjepan's comments are not that he wants to 
require a manager to take on a huge workload, but he would like to be 
sure there is some inspection before the review is scheduled. If I 
misunderstood him, I hope he will correct me.

   My personal opinion is that this position is reasonable. If the 
manager is an experienced member of the boost community, then we can 
expect some familiarity with the quality standards. If there is some 
reason the library clearly doesn't meet those standards, then the 
manager should know that before scheduling, and take appropriate action. 
If there is no clear deficiency, the review should proceed and let the 
reviewers reach their own conclusions.

   I know full well that managing a review is a large time commitment, 
and I'm very sensitive to anyone who is concerned that it not become 
overwhelming. If the general opinion is that this is too much, then we 
won't do it. However, I want the reviewers to feel that studying the 
library and producing a review is a valuable use of their time, too. 
Making sure that libraries are ready for review before scheduling is a 
necessary step toward convincing them.

                        John