Subject: Re: [boost] is review system in place is extremely slow? (was Re:[rfc] rcpp)
From: John Phillips (phillips_at_[hidden])
Date: 2010-03-01 23:02:01
Gennadiy Rozental wrote:
> Vicente Botet wrote:
>> This should be checked already by the review manager.
> I guess. I was after a script that can perform basis automatic checks.
I don't know of any managers using them, but I think many such checks
already can be done with scripts we have and use for other things. The
script that checks for license inclusion and other such things in the
release libraries could just as easily be run on a submission, after all.
>>> Once review manager is assigned candidate page is transformed into
>>> "candidate review" page.
>> Who will assign the review manager?
> The review wizards. Forgot the term.
>>> 7. If there is no review manager found within a year, the library is
>>> rejected due to lack of support.
>> This seems OK to me. A clarification of who can be the review manager
>> will be needed.
> There are two parts of review manager job, which in fact can be split.
> The first part is actually managing the review, including the checking
> of the formal requirements, making announcements, producing summaries
> (as per my suggestion), making decision about review start, review
> length, review stop. The second part is making the final decision.
YMMV, but for me the administrative part of the job is a small
commitment. The technical part of the job is what takes time. I place
the production of useful summaries in the technical side.
> As I see it the first part does not require any particular experience
> and can be delegated to any one (even relative newcomer or someone
> nominated by library author). The second part is where we expect to have
> some authority in the matter. If review results are overwhelmingly
> positive or negative (like acceptance by 95% of reviewers, or rejection
> by more than 50-60%) the review result decision can be done automatically.
I'm not a big fan of a straight vote based cutoff. (I'm speaking as a
Boost member, not as a Review Wizard when expressing such opinions.) If
20 people vote in favor of a library, but one person with a show
stopping reason for a rejection is right that it is a big deal issue,
then the library should be rejected. If half the respondents object to a
library, but there is an important flaw in their reasons for that
objection, then it might be a good idea to accept the library.
I think of the votes as a well intentioned advisory body, not a
definitive response. The Manager should work to understand the reasons
for those votes, and then weight the reasons instead of the votes as the
method to reach a decision.
> In more complicated case we need another person(s) of authority. One
> option I see here is for the review manager (one who performs first
> step) to write detailed summary with final tally of votes and all major
> pro and con discussion points and allow review wizard(s) to make this
> final decision (or some new post, like review arbitrager).
If this lesser manager is not technically competent to make the
decision, why should anyone believe there is competence to write a
summary that provides all of the important points in the discussion with
appropriate justifications? It is harder to write such a summary than to
understand the arguments, after all.
Even given such a summary, I strongly doubt that any one person has
the breadth and depth of expertise to be able to make such judgments for
every library that might be proposed for Boost. I openly state that I
claim no such development omniscience.
As I always say in these discussions, my preference is to trust the
review managers. Put someone in the post who will look carefully at the
arguments during the review and reason carefully about them. Monitor the
reviews to make sure things stay on track (I read every post in every
review in Boost as part of being a Wizard, to be sure I know what is
happening in them. I don't like surprises in the review process.), and
let the managers apply their skills to come to a good conclusion.
It is imaginable that a manager could make an unsupportable decision.
If this happens, then the wizards and the monitors will have to step in
and determine the best course of action. This has not happened while I
have been a Wizard, and I know of no instance before I was a Wizard,
either. I sincerely hope it doesn't happen, but will act as needed if it
does. Thankfully, we have had some very good people acting as managers
through the years, and I appreciate how much easier they have made my job.
Boost list run by bdawes at acm.org, gregod at cs.rpi.edu, cpdaniel at pacbell.net, john at johnmaddock.co.uk