Re: [boost] is review system in place is extremely slow? (was Re: [rfc] rcpp)

Subject: Re: [boost] is review system in place is extremely slow? (was Re: [rfc] rcpp)
From: Gennadiy Rozental (rogeeff_at_[hidden])
Date: 2010-02-28 11:24:47

Andrey Semashev wrote:
> Hi Gennadiy,
> 
> While I mostly agree with you pointing out the drawbacks of the current 
> system, I don't quite agree with your proposal.
> 
> On 02/28/2010 04:24 PM, Gennadiy Rozental wrote:
>>
>> That's said, here's how better procedure might look like IMO. This will
>> require some initial investment in writing scripts for process
>> automation, but in a long run we should be very well compensated.
>>
>> 1. Any library author interrested in submission of new library should
>> come to the "Candidate" page and register. Once registered candidate 
>> gets:
>> a) svn repository for the library
>> b) standardized page on boost website (something like
>> boost.org/candidate/<candidate name>
>> c) announcement post is sent automatically (with abstract and link to
>> above page) to the mailing list.
> 
> Good. Having a central place for potential Boost libraries to evolve may 
> simplify development. Although I'm not sure there are resources to 
> maintain this kind of hosting.

We already hosting most of the library wait in svn. Candidate page 
should be comparatively tiny and only contains abstracts and links.

>> 2. The candidate page should contain abstract and links to the sources
>> and docs. Also it should include some kind of "voting" mechanism, where
>> people would express the interest. Preferable with authentication, which
>> would link to the mailing list members. To qualify for the review
>> candidate should exceed some predefined threshold of minimum number of
>> "supporters". These people are expected to post a review later on for
>> the library to have a chance of being accepted.
> 
> Voting is good. I appreciated the feature on SourceForge. Although I 
> don't think that the right to vote should be tied with posting a review 
> later. I consider voting as afeedback mechanism, nothing more.

I am not gonna push this point. IMO it's fair to expect people who 
seconded the submission to provide a review within rather extended 
review period. On the other hand we can't really put a "requirement" 
clause to the support vote.

> Regarding the candidate page, do you mean that the library docs should 
> be hosted somewhere outside the Boost web site? If so, I don't like that 
> idea. IMO, if we pursue the idea of a central hosting for the candidate 
> libraries (with SVN, web access, etc.), it should include online 
> documentation hosting, too.

No. I think the link should be somewhere inside the svn, or be the part 
of candidate page. If former case we need a script+link which extracts 
docs from svn, in later case - overhead is bigger.

>> 3. Once candidate have proper number of supporters and passed all other
>> formal requirements (docs, tests, directory structure) - all validated
>> against repository, candidate author can schedule a review from reviews
>> schedulers (whatever the proper name). Once review manager is assigned
>> candidate page is transformed into "candidate review" page.
> 
> It's not clear how it's transformed and in what way. Regarding the 
> review scheduling, it's pretty much like it happens nowdays.

Not really. The transition occurs almost automatically and 
asynchronously. There is no any queue. If one library has bigger support 
and interest in a community, it will go through the phases much faster.

>> 4. Review process.
>> The candidate review can start at any time by the review manager (no
>> queue) and should take at least 2-4 month. There can be any number of
>> reviewed being run concurrently. The "candidate review" page should
>> include abstract, review package, and some kind of review submission
>> mechanism (maybe boolean yes/no + an actual review). The review should
>> be per person and each reviewer should have an ability to modify the
>> review.
>> Review discussion mechanism can be web based on rely on mailing list or
>> some mixture of these.
> 
> I disagree, in several points.
> 
> * 2-4 months is a very long period. 

Somehow you are fine with C++ standard changes being reviewed for years, 
but find 2-4 month for non-trivial C++ library too long?

> You can't expect review manager and 
> the library authors focused on the review that long. 

Actually the point was to decrease the pressure. Longer time period 
means that both reviewers and author can take their time doing their 
job. I expect short short period of times with high activity with some 
gaps in between, where sides consider the matter.

> Also, for simple 
> tools, such as Boost.Move that is in the queue now, there's nothing to 
> review during all that time. 

For smaller/simple components we might have a policy of fast track 
review (no more than a week. These we can have a queue for, but there 
should be rather strict requirements to qualify:
* 1 header only (or totally no more than N lines). Header only libraries 
qualify
* All tests should work on all required platforms
* Docs should be in boostbook format
* To be accepted the library should receive 90%(?) approval within 
review period with at least 10(?) reviewers
* Review period is never extended

> On the other hand, I agree that a few weeks 
> may not be enough for some larger scaled libraries. Which leads me to 
> conclusion that the review duration should be individual, decided by the 
> author, review manager and review wizards, taking into account other 
> reviews.

Ultimately yes, but only if library qualify for fast track review. 
Otherwise I do not see basis for review wizard to warrant it.

> * Concurrent reviews is wrong. We don't have enough reviewers and 
> wizards to make sequential reviews. Allowing parallel reviews won't make 
> it better. The review quality will also drop.

Even in rare situation where the same person is interested in several 
concurrent reviews, long review period should give one a chance to 
participate in both.

> * Review mechanism should be convenient for both the reviewers and the 
> author/review manager. It should allow an easy conversation between the 
> reviewers and the author. Mailing list is good enough, I think.

IMO review mechanism can be much better. Ideally coming to the one 
should have a change to get a summary of current discussions, open 
issues and author comments to them. Not sure if we can automate the 
procedure or it should be a review manager job. Maybe we can combine 
active discussion going on mailing list with summaries appearing on 
candidate review page.

>> 5. Review manager have a right to stop a review at any time and make a
>> decision if there is an overwhelming evidence that the library is going
>> to be accepted/rejected.
> 
> Ok.
> 
>> 6. If there is not enough reviews with first 2-4 month, the library is
>> rejected due to lack of support.
> 
> Hmm, arguable, at least. If it made it to the review, there surely is an 
> interest to the library.

There maybe number of reasons. It possible that the final product did 
not meet the expectations or original supported disappeared and no one 
else come through. In any case there is no reason to expect the 
situation will change. IMO in this case library can do back to the 
candidate phase and wait for new set of supporters.

>> 7. If there is no review manager found within a year, the library is
>> rejected due to lack of support.
> 
> I think, there are several useful libraries in the queue that fit that 
> criteria. My Boost.Log was surely longer than a year without a review 
> manager, and I can't say there's no interest.

I do not believe the Boost.Log waiting for review solely due to the lack 
of the review manager. I managed last one. And I believe there were 
couple candidates for the job for new take.

> For both 6 and 7, bouncing candidates away won't help the situation.
> 
> 
> And the most important objection from my side is that your proposal 
> doesn't change anything to solve the root problem - there are not enough 
> people (or free time of theirs) to manage and write full reviews. I 
> actually makes it a bit worse.

IMO :

1. It should allow more people to write full review given more time and 
less pressure to do it within short period of time

2. It should allow more libraries to go through the review process 
ultimately, due to concurrent nature of the process.

3. It should make more people willing to manage the review, given that 
procedure does not require paying significant attention to the library 
within the review period.

Gennadiy