Re: [boost] [review queue] What to do about the library review queue?

Subject: Re: [boost] [review queue] What to do about the library review queue?
From: Hartmut Kaiser (hartmut.kaiser_at_[hidden])
Date: 2017-03-14 14:31:50

> I see that new candidate Boost libraries entering the review queue have
> exploded in recent years, with no less than *twenty-three* proposed
> libraries awaiting a review.
> 
> As the ongoing strength and vitality of Boost is inextricably linked to
> new growth, I think that waiting around for years for someone to
> volunteer to manage a review is not healthy. If a library author has
> invested the very significant effort to develop a Boost-quality library,
> the least Boost can do is to try harder to provide timely reviews and
> that means persuading more people to volunteer to manage reviews.
> 
> In the past people have argued that for every library you submit for
> review, you should manage a review in return. Myself, Antony and a few
> others have adhered to that rule, and if every library author did so
> there would be no outstanding review queue. However there are problems
> in that in itself in terms of moral hazard, and also because the review
> manager needs to usually be fairly expert in a library being reviewed,
> else it can be very hard to judge the worth and validity of reviews. A
> shortage of suitably expert review managers will always be a problem for
> some types of library.
> 
> 
> I therefore ask boost-dev what to do? Some options:
> 
> 1. Pay US$1000 (one thousand) dollars to each person who manages a
> review. In case you're worried Boost doesn't have the money, it does in
> spades, that's not a problem. For $23,000 we could clear the current
> review queue assuming none of the problems mentioned yet.
> 
> 2. Pay US$1000 dollars to the manager and 2x $500 dollar payments to
> those writing the top two most useful reviews as judged by the review
> manager. That makes the cost $2000 per library accepted or rejected, or
> $46,000 to clear the current review queue.
> 
> 3. In my own opinion from reviewing the review queue, a good 25% of the
> libraries in the queue are not ready for review due to obvious glaring
> deficiencies in the documentation or code. Spending a grand on those
> libraries which will very obviously be rejected isn't worth the money.
> What should we do about those? One approach could simply be to trust
> review managers to not abuse the thousand dollar fee. Another could be
> that before each new review, the prospective manager needs to write a
> single line comment on why they did not choose the other libraries in
> the queue and publish that here before starting a review. That would
> quickly identify those libraries in the queue which a majority of
> managers think have serious problems and could never pass any review. If
> say a library in a queue accumulates three single line black marks, the
> author might be encouraged to withdraw it.
> 
> 4. Finally there is the problem of libraries of high quality, but not a
> good fit for Boost because they are so esoteric and niche that nobody
> could provide a useful review, and without useful reviews the review
> manager can't really recommend acceptance. This will be an increasing
> problem with time anyway as more of the low hanging C++ library fruit is
> picked, but I suppose one could just kick that decision can down the
> road and see if 2x $500 payments might help scare up more high quality
> reviews.
> 
> 5. We could try guilting more people into review managing, and redouble
> banging the drum to scare up more volunteers.

Let me be upfront. I'm strictly against correlating the review process with
money in any way.

Boost's mission is to foster the development of widely applicable, high
quality, and freely available C++ libraries. It also provides a uniform
platform allowing to distribute those libraries. Over the years it has
maintained a widely accepted quality standard which makes it desirable for
people to submit libraries and to become Boost library authors.

None of these points involve the need to increase the amount of accepted
libraries at any cost. As others have pointed out, Boost's livelihood does
not directly depend on the number of accepted libraries but just on their
quality. Adding money to the mix favors adding libraries at all cost,
non-withstanding their quality or real-world-need, emphasizing the
commercial aspect.

Having the review process being volunteer-driven guarantees a) a real-world
need for the library under review, b) fairness of the decision, c) a high
quality of the review, d) direct interest in organizing the review by the
review manager (otherwise he/she wouldn't do it).

Starting to add money to the volunteer-process of reviewing libraries will
set off the balance between quality and usability. It increases the review
managers interest in doing a review (point (d) from the list above), but
directly diminishes the importance of (a), (b), and (c). 

Letting Boost pay for reviews implies that Boost as an organization is
interested in adding as many libraries as possible, regardless of real
interest in the community in a particular library. I think this is not in
the interest of the Boost organization.

Adding money to the process will immediately destroy the transparency of the
review process and it most likely will be detrimental to its quality. Both
are things we should avoid by all means possible.

This would also raise a lot of questions I wouldn't even know how to start
answering. Like:

- What's next? Letting library authors pay for their library being reviewed?
After all THEY are the most interested parties in adding their work to
Boost...
- Or perhaps accepting 'donations' from companies earmarked for paying a
review manager, further skewing the review process?
- Will every review manager receive the money? Regardless of the quality of
how the review is managed? What would be the criteria for a review manager
doing a good enough job to receive the payment? Who decides on this? How
many reviews would be a single person be allowed to perform? 
- Would previous review managers receive an equally generous payment for the
libraries reviewed in the past? If yes - why, if no - why not?
- and many more...

In short, while I understand what issues this proposal is trying to solve, I
strongly believe it chooses inadequate means of doing so.

Regards Hartmut
---------------
http://boost-spirit.com
http://stellar.cct.lsu.edu