Re: [boost] is review system in place is extremely slow? (was Re:[rfc] rcpp)

Subject: Re: [boost] is review system in place is extremely slow? (was Re:[rfc] rcpp)
From: John Phillips (phillips_at_[hidden])
Date: 2010-03-01 23:02:01

Gennadiy Rozental wrote:
> Vicente Botet wrote:
>>
>> This should be checked already by the review manager.
> 
> I guess. I was after a script that can perform basis automatic checks.

   I don't know of any managers using them, but I think many such checks 
already can be done with scripts we have and use for other things. The 
script that checks for license inclusion and other such things in the 
release libraries could just as easily be run on a submission, after all.

> 
>>> Once review manager is assigned candidate page is transformed into 
>>> "candidate review" page.
>>
>> Who will assign the review manager?
> 
> The review wizards. Forgot the term.
> 
>>> 7. If there is no review manager found within a year, the library is 
>>> rejected due to lack of support.
>>
>> This seems OK to me. A clarification of who can be the review manager 
>> will be needed.
> 
> There are two parts of review manager job, which in fact can be split.
> 
> The first part is actually managing the review, including the checking 
> of the formal requirements, making announcements, producing summaries 
> (as per my suggestion), making decision about review start, review 
> length, review stop. The second part is making the final decision.

   YMMV, but for me the administrative part of the job is a small 
commitment. The technical part of the job is what takes time. I place 
the production of useful summaries in the technical side.

> 
> As I see it the first part does not require any particular experience 
> and can be delegated to any one (even relative newcomer or someone 
> nominated by library author). The second part is where we expect to have 
> some authority in the matter. If review results are overwhelmingly 
> positive or negative (like acceptance by 95% of reviewers, or rejection 
> by more than 50-60%) the review result decision can be done automatically.

   I'm not a big fan of a straight vote based cutoff. (I'm speaking as a 
Boost member, not as a Review Wizard when expressing such opinions.) If 
20 people vote in favor of a library, but one person with a show 
stopping reason for a rejection is right that it is a big deal issue, 
then the library should be rejected. If half the respondents object to a 
library, but there is an important flaw in their reasons for that 
objection, then it might be a good idea to accept the library.

   I think of the votes as a well intentioned advisory body, not a 
definitive response. The Manager should work to understand the reasons 
for those votes, and then weight the reasons instead of the votes as the 
method to reach a decision.

> 
> In more complicated case we need another person(s) of authority. One 
> option I see here is for the review manager (one who performs first 
> step) to write detailed summary with final tally of votes and all major 
> pro and con discussion points and allow review wizard(s) to make this 
> final decision (or some new post, like review arbitrager).
> 
> Gennadiy
> 

   If this lesser manager is not technically competent to make the 
decision, why should anyone believe there is competence to write a 
summary that provides all of the important points in the discussion with 
appropriate justifications? It is harder to write such a summary than to 
understand the arguments, after all.

   Even given such a summary, I strongly doubt that any one person has 
the breadth and depth of expertise to be able to make such judgments for 
every library that might be proposed for Boost. I openly state that I 
claim no such development omniscience.

   As I always say in these discussions, my preference is to trust the 
review managers. Put someone in the post who will look carefully at the 
arguments during the review and reason carefully about them. Monitor the 
reviews to make sure things stay on track (I read every post in every 
review in Boost as part of being a Wizard, to be sure I know what is 
happening in them. I don't like surprises in the review process.), and 
let the managers apply their skills to come to a good conclusion.

   It is imaginable that a manager could make an unsupportable decision. 
If this happens, then the wizards and the monitors will have to step in 
and determine the best course of action. This has not happened while I 
have been a Wizard, and I know of no instance before I was a Wizard, 
either. I sincerely hope it doesn't happen, but will act as needed if it 
does. Thankfully, we have had some very good people acting as managers 
through the years, and I appreciate how much easier they have made my job.

                        John Phillips
                        Review Wizard