Re: [boost] is review system in place is extremely slow? (was Re:[rfc] rcpp)

Subject: Re: [boost] is review system in place is extremely slow? (was Re:[rfc] rcpp)
From: Vicente Botet (vicente.botet_at_[hidden])
Date: 2010-02-28 16:05:47

Hi Gennadiy,

first of all thenkw for trying to setup some improvements to the review process.

----- Original Message ----- 
From: "Gennadiy Rozental" <rogeeff_at_[hidden]>
To: <boost_at_[hidden]>
Sent: Sunday, February 28, 2010 2:24 PM
Subject: Re: [boost] is review system in place is extremely slow? (was Re:[rfc] rcpp)

> 
> vicente.botet wrote:
>> Why do you think the review system in place is extremely slow? 
>> 
>> Currently there are a lot of libraries to review, but no review managers. That means that the user community don't want to spend a little bit of their time to manage a review.
>> 
>> In addition, the last review didn't had too much of reviewers (I'm also concerned by this point)
> 
> 
> Here are my 2 cents. I've expressed this opinion previously, but nothing 
> really changed since then.
> 
> IMHO The review system is both too slow and too fast. Why it's too slow:
> * Not enough review managers
> * Not enough reviewers - reviews keep being extended
> * Not enough reviews per year. Too many limiting factors, like holidays, 
> upcoming or just completed releases

Agree.
 
> Why it's too fast:
> 
> * IMHO any short period of time is too short to properly evaluate most 
> non-trivial libraries
>
> * To accumulate proper number of non-trivial reviews usually require 
> time for people who are not regular on a mailing list to actually come 
> and see that there is one. Also take into an account that we are loosing 
> people who are for whatever reason indispensable during scheduled review
> 
> * Some libraries come up without proper substantiation leading to review 
> process and only being rejected by "lack of interest" argument

The author and the review manager should start already a review only if they have checked that the library has enough interest..
 
> * Some libraries comes not being ready for review. There is 
> automatically checked list of requirements before scheduling the review.

This should be checked already by the review manager.
 
> That's said, here's how better procedure might look like IMO. This will 
> require some initial investment in writing scripts for process 
> automation, but in a long run we should be very well compensated.
> 
> 1. Any library author interrested in submission of new library should 
> come to the "Candidate" page and register. Once registered candidate gets:
>    a) svn repository for the library
>    b) standardized page on boost website (something like 
> boost.org/candidate/<candidate name>
>    c) announcement post is sent automatically (with abstract and link 
> to above page) to the mailing list.

If I've understood you are proposing a separate space from the sandbox. Isn't it?

I agree that it is better if people that pretend to be candidate to Boost use to use the Boost environment and tools.

> 2. The candidate page should contain abstract and links to the sources 
> and docs. Also it should include some kind of "voting" mechanism, where 
> people would express the interest. Preferable with authentication, which 
> would link to the mailing list members. To qualify for the review 
> candidate should exceed some predefined threshold of minimum number of 
> "supporters". These people are expected to post a review later on for 
> the library to have a chance of being accepted.

This seems good. If people wants a library in Boost they must support the library at least engaging themself to make a review.

> 3. Once candidate have proper number of supporters and passed all other 
> formal requirements (docs, tests, directory structure) - all validated 
> against repository, 

> candidate author can schedule a review from reviews 
> schedulers (whatever the proper name). 

I don't understand this. Could you clarify?

> Once review manager is assigned 
> candidate page is transformed into "candidate review" page.

Who will assign the review manager?
 
> 4. Review process.
> The candidate review can start at any time by the review manager (no 
> queue) and should take at least 2-4 month. There can be any number of 
> reviewed being run concurrently. The "candidate review" page should 
> include abstract, review package, and some kind of review submission 
> mechanism (maybe boolean yes/no + an actual review). The review should 
> be per person and each reviewer should have an ability to modify the 
> review.

I agree with the extension of the review period.

> Review discussion mechanism can be web based on rely on mailing list or 
> some mixture of these.
> 
> 5. Review manager have a right to stop a review at any time and make a 
> decision if there is an overwhelming evidence that the library is going 
> to be accepted/rejected.

People wanting the library to be included should be active supporters and don't expect until the end of the review period, an extension of the period should not correspond to an inactive period. Thus I agree.
 
> 6. If there is not enough reviews with first 2-4 month, the library is 
> rejected due to lack of support.

I will extend this period to 4-6month.
 
> 7. If there is no review manager found within a year, the library is 
> rejected due to lack of support.

This seems OK to me. A clarification of who can be the review manager will be needed.
 
> Time periods here are tentative are subject for discussion. Also 
> specific collaboration between candidate review page and mailing list 
> need clarification.

Thanks,
Vicente