Re: [boost] is review system in place is extremely slow? (was Re: [rfc] rcpp)

Subject: Re: [boost] is review system in place is extremely slow? (was Re: [rfc] rcpp)
From: Gennadiy Rozental (rogeeff_at_[hidden])
Date: 2010-02-28 08:24:31

vicente.botet wrote:
> Why do you think the review system in place is extremely slow? 
> 
> Currently there are a lot of libraries to review, but no review managers. That means that the user community don't want to spend a little bit of their time to manage a review.
> 
> In addition, the last review didn't had too much of reviewers (I'm also concerned by this point)

Here are my 2 cents. I've expressed this opinion previously, but nothing 
really changed since then.

IMHO The review system is both too slow and too fast. Why it's too slow:
* Not enough review managers
* Not enough reviewers - reviews keep being extended
* Not enough reviews per year. Too many limiting factors, like holidays, 
upcoming or just completed releases

Why it's too fast:

* IMHO any short period of time is too short to properly evaluate most 
non-trivial libraries

* To accumulate proper number of non-trivial reviews usually require 
time for people who are not regular on a mailing list to actually come 
and see that there is one. Also take into an account that we are loosing 
people who are for whatever reason indispensable during scheduled review

* Some libraries come up without proper substantiation leading to review 
process and only being rejected by "lack of interest" argument

* Some libraries comes not being ready for review. There is 
automatically checked list of requirements before scheduling the review.

That's said, here's how better procedure might look like IMO. This will 
require some initial investment in writing scripts for process 
automation, but in a long run we should be very well compensated.

1. Any library author interrested in submission of new library should 
come to the "Candidate" page and register. Once registered candidate gets:
    a) svn repository for the library
    b) standardized page on boost website (something like 
boost.org/candidate/<candidate name>
    c) announcement post is sent automatically (with abstract and link 
to above page) to the mailing list.

2. The candidate page should contain abstract and links to the sources 
and docs. Also it should include some kind of "voting" mechanism, where 
people would express the interest. Preferable with authentication, which 
would link to the mailing list members. To qualify for the review 
candidate should exceed some predefined threshold of minimum number of 
"supporters". These people are expected to post a review later on for 
the library to have a chance of being accepted.

3. Once candidate have proper number of supporters and passed all other 
formal requirements (docs, tests, directory structure) - all validated 
against repository, candidate author can schedule a review from reviews 
schedulers (whatever the proper name). Once review manager is assigned 
candidate page is transformed into "candidate review" page.

4. Review process.
The candidate review can start at any time by the review manager (no 
queue) and should take at least 2-4 month. There can be any number of 
reviewed being run concurrently. The "candidate review" page should 
include abstract, review package, and some kind of review submission 
mechanism (maybe boolean yes/no + an actual review). The review should 
be per person and each reviewer should have an ability to modify the 
review.
Review discussion mechanism can be web based on rely on mailing list or 
some mixture of these.

5. Review manager have a right to stop a review at any time and make a 
decision if there is an overwhelming evidence that the library is going 
to be accepted/rejected.

6. If there is not enough reviews with first 2-4 month, the library is 
rejected due to lack of support.

7. If there is no review manager found within a year, the library is 
rejected due to lack of support.

Time periods here are tentative are subject for discussion. Also 
specific collaboration between candidate review page and mailing list 
need clarification.