Subject: Re: [boost] is review system in place is extremely slow? (was Re: [rfc] rcpp)
From: Gennadiy Rozental (rogeeff_at_[hidden])
Date: 2010-02-28 11:24:47
Andrey Semashev wrote:
> Hi Gennadiy,
> While I mostly agree with you pointing out the drawbacks of the current
> system, I don't quite agree with your proposal.
> On 02/28/2010 04:24 PM, Gennadiy Rozental wrote:
>> That's said, here's how better procedure might look like IMO. This will
>> require some initial investment in writing scripts for process
>> automation, but in a long run we should be very well compensated.
>> 1. Any library author interrested in submission of new library should
>> come to the "Candidate" page and register. Once registered candidate
>> a) svn repository for the library
>> b) standardized page on boost website (something like
>> boost.org/candidate/<candidate name>
>> c) announcement post is sent automatically (with abstract and link to
>> above page) to the mailing list.
> Good. Having a central place for potential Boost libraries to evolve may
> simplify development. Although I'm not sure there are resources to
> maintain this kind of hosting.
We already hosting most of the library wait in svn. Candidate page
should be comparatively tiny and only contains abstracts and links.
>> 2. The candidate page should contain abstract and links to the sources
>> and docs. Also it should include some kind of "voting" mechanism, where
>> people would express the interest. Preferable with authentication, which
>> would link to the mailing list members. To qualify for the review
>> candidate should exceed some predefined threshold of minimum number of
>> "supporters". These people are expected to post a review later on for
>> the library to have a chance of being accepted.
> Voting is good. I appreciated the feature on SourceForge. Although I
> don't think that the right to vote should be tied with posting a review
> later. I consider voting as afeedback mechanism, nothing more.
I am not gonna push this point. IMO it's fair to expect people who
seconded the submission to provide a review within rather extended
review period. On the other hand we can't really put a "requirement"
clause to the support vote.
> Regarding the candidate page, do you mean that the library docs should
> be hosted somewhere outside the Boost web site? If so, I don't like that
> idea. IMO, if we pursue the idea of a central hosting for the candidate
> libraries (with SVN, web access, etc.), it should include online
> documentation hosting, too.
No. I think the link should be somewhere inside the svn, or be the part
of candidate page. If former case we need a script+link which extracts
docs from svn, in later case - overhead is bigger.
>> 3. Once candidate have proper number of supporters and passed all other
>> formal requirements (docs, tests, directory structure) - all validated
>> against repository, candidate author can schedule a review from reviews
>> schedulers (whatever the proper name). Once review manager is assigned
>> candidate page is transformed into "candidate review" page.
> It's not clear how it's transformed and in what way. Regarding the
> review scheduling, it's pretty much like it happens nowdays.
Not really. The transition occurs almost automatically and
asynchronously. There is no any queue. If one library has bigger support
and interest in a community, it will go through the phases much faster.
>> 4. Review process.
>> The candidate review can start at any time by the review manager (no
>> queue) and should take at least 2-4 month. There can be any number of
>> reviewed being run concurrently. The "candidate review" page should
>> include abstract, review package, and some kind of review submission
>> mechanism (maybe boolean yes/no + an actual review). The review should
>> be per person and each reviewer should have an ability to modify the
>> Review discussion mechanism can be web based on rely on mailing list or
>> some mixture of these.
> I disagree, in several points.
> * 2-4 months is a very long period.
Somehow you are fine with C++ standard changes being reviewed for years,
but find 2-4 month for non-trivial C++ library too long?
> You can't expect review manager and
> the library authors focused on the review that long.
Actually the point was to decrease the pressure. Longer time period
means that both reviewers and author can take their time doing their
job. I expect short short period of times with high activity with some
gaps in between, where sides consider the matter.
> Also, for simple
> tools, such as Boost.Move that is in the queue now, there's nothing to
> review during all that time.
For smaller/simple components we might have a policy of fast track
review (no more than a week. These we can have a queue for, but there
should be rather strict requirements to qualify:
* 1 header only (or totally no more than N lines). Header only libraries
* All tests should work on all required platforms
* Docs should be in boostbook format
* To be accepted the library should receive 90%(?) approval within
review period with at least 10(?) reviewers
* Review period is never extended
> On the other hand, I agree that a few weeks
> may not be enough for some larger scaled libraries. Which leads me to
> conclusion that the review duration should be individual, decided by the
> author, review manager and review wizards, taking into account other
Ultimately yes, but only if library qualify for fast track review.
Otherwise I do not see basis for review wizard to warrant it.
> * Concurrent reviews is wrong. We don't have enough reviewers and
> wizards to make sequential reviews. Allowing parallel reviews won't make
> it better. The review quality will also drop.
Even in rare situation where the same person is interested in several
concurrent reviews, long review period should give one a chance to
participate in both.
> * Review mechanism should be convenient for both the reviewers and the
> author/review manager. It should allow an easy conversation between the
> reviewers and the author. Mailing list is good enough, I think.
IMO review mechanism can be much better. Ideally coming to the one
should have a change to get a summary of current discussions, open
issues and author comments to them. Not sure if we can automate the
procedure or it should be a review manager job. Maybe we can combine
active discussion going on mailing list with summaries appearing on
candidate review page.
>> 5. Review manager have a right to stop a review at any time and make a
>> decision if there is an overwhelming evidence that the library is going
>> to be accepted/rejected.
>> 6. If there is not enough reviews with first 2-4 month, the library is
>> rejected due to lack of support.
> Hmm, arguable, at least. If it made it to the review, there surely is an
> interest to the library.
There maybe number of reasons. It possible that the final product did
not meet the expectations or original supported disappeared and no one
else come through. In any case there is no reason to expect the
situation will change. IMO in this case library can do back to the
candidate phase and wait for new set of supporters.
>> 7. If there is no review manager found within a year, the library is
>> rejected due to lack of support.
> I think, there are several useful libraries in the queue that fit that
> criteria. My Boost.Log was surely longer than a year without a review
> manager, and I can't say there's no interest.
I do not believe the Boost.Log waiting for review solely due to the lack
of the review manager. I managed last one. And I believe there were
couple candidates for the job for new take.
> For both 6 and 7, bouncing candidates away won't help the situation.
> And the most important objection from my side is that your proposal
> doesn't change anything to solve the root problem - there are not enough
> people (or free time of theirs) to manage and write full reviews. I
> actually makes it a bit worse.
1. It should allow more people to write full review given more time and
less pressure to do it within short period of time
2. It should allow more libraries to go through the review process
ultimately, due to concurrent nature of the process.
3. It should make more people willing to manage the review, given that
procedure does not require paying significant attention to the library
within the review period.
Boost list run by bdawes at acm.org, gregod at cs.rpi.edu, cpdaniel at pacbell.net, john at johnmaddock.co.uk