Re: [boost] [GSoC] [Boost.Hana] Formal review request

Subject: Re: [boost] [GSoC] [Boost.Hana] Formal review request
From: Niall Douglas (s_sourceforge_at_[hidden])
Date: 2014-07-29 20:00:11

On 29 Jul 2014 at 17:02, Louis Dionne wrote:

> > Also, regarding formal review, I personally would feel uncomfortable 
> > accepting a library that only works with a single version of clang. I 
> > would feel much happier if you got trunk GCC working, even if that 
> > means workarounds.
> 
> That would mean a lot of workarounds.

Not necessarily. You could wait until GCC trunk catches up instead, 
helping it along by adding however many bug reports as is necessary. 
Timing well a formal review is often as important as the quality of 
your library.

I always feel heebie jeebies about code which works only on one 
compiler. For me to vote yes for a Boost library to enter Boost I 
need to feel it is well tested and reliable and has had all its kinks 
knocked out. I struggle to see me feeling this with a library which 
can't be tested widely.

Instead of heading straight into the community review queue, perhaps 
a few rounds of intermediate informal reviews like this one? I'd 
particularly like to see Eric and Joel's opinion of your library so 
far too.

> Basically, I got a benchmark for almost every operation of almost every 
> sequence that's supported by Hana (including those adapted from external 
> libraries), but I'm not sure yet of how to group them in the documentation
> (per operation? per sequence? per type class?).
> 
> The problem is made worse by two things:
> 
>   - It only makes sense to benchmark components that are isomorphic. For 
>     example, what does it mean to benchmark a std::tuple against a 
>     mpl::vector? Not much, because the price you pay for std::tuple 
>     gives you the ability to hold values, whereas mpl::vector can only 
>     hold types. We don't want to compare apples with oranges, and the 
>     grouping of benchmarks should be influenced by that.

I guess it's like when choosing between list, map and vector from the 
STL. All are "equivalent", but you need to know how each might scale 
out for what you need. Most users won't just known automatically.

My problem always with MPL98 was I had no idea what was fast or slow 
for my use cases.

>   - How do we handle different compilers? Right now, all benchmarks are
>     produced only on Clang, which is OK because it's the only compiler that
>     can compile the library. When there is more than one compiler, how do we
>     generate the benchmarks for all of them, and how do we integrate the
>     benchmarks in the documentation?

You're right this is more of a name and shame thing between AST 
compilers. MSVC is very different though, and you may eventually need 
a separate line labelled "MSVC" just for it.

> > I'd also like to see unit testing that verified that the current 
> > compiler being tested has a time and space benchmark curve matching 
> > what is expected. It is too easy for code to slip in or the compilers 
> > themselves to gain a bug which creates pathological metaprogramming 
> > performance. Better to have Travis CI trap that for you than head 
> > scratching and surprises later.
> 
> I thought about doing this, but I did not because I thought it was a HUGE 
> undertaking to automate it.

No, it's easier than you think. Have a look at 
https://ci.nedprod.com/ whose default dashboard shows a graph 
labelled "RUDP performance". This tracks performance of a build over 
time to ensure performance doesn't regress. All you need is for your 
performance test tool to output some CSV, a Jenkins Plot plugin does 
the rest.

> I think we're better off integrating the 
> benchmarks in the documentation and then when something is really weird,
> we just have to look at the generated benchmarks and see what's wrong.
> If someone can suggest a way to do it automatically that won't take me weeks
> to set up, I'm interested.

Mastering Jenkins takes months, but once mastered configuring all 
sorts of test scenarios becomes trivial. I'd actively merge your 
Jenkins/Travis output into your docs too, it is nowadays an online 
world.

> Also, I wouldn't want to slip in the trap of testing the compiler; testing
> Hana is a large enough task as it is (341 tests + 165 examples as we speak).

I think testing modern C++ is always testing the compiler. It's why I 
develop first on MSVC, I stand a good chance of porting that to GCC 
or clang, less so the other way round.

> > Finally, there are ways and means for doxygen docs to automatically 
> > convert into BoostBook docs. You'll need to investigate those before 
> > starting a formal review. Tip: look into how Geometry/AFIO does the 
> > doxygen conversion, it's brittle but it is easier than the others.
> 
> Is it mandatory for a Boost library to have BoostBook documentation?
> I'd like to stay as mainstream as possible in the tools I use and reduce
> the number of steps in the build/documentation process for the sake of
> simplicity. Is there a gain in generating the documentation in BoostBook?

As you'll find when formal review comes, to pass isn't about how good 
your library is, it's about eliminating as many rational objections 
others can think of.

BoostBook format documentation is a very easy way of crossing 
permanently off a raft of potential objections. Other categories of 
objection will be far harder to address, trust me. I suspect the 
hardest will probably be the "so what?" objection in all the guises 
it manifests, as it always is for any library which is particularly 
novel.

All that said, I have little love for the Boost documentation format, 
I think it outdated, brittle, and inflexible, and it does a lousy job 
of automating reference documentation. But it could be a lot worse, 
and I have been forced in the past to use a lot worse.

Besides, you'll invest five days or so of wishing pain on those 
responsible for the tools, and once it's working you'll never need 
touch it again. I did find it took some months to find and fix all 
the corner cases in the doc output though, and even now PDF 
generation from AFIO's docs are a joke due to the long template 
strings.

Anyway, it's up to you. BTW, I've noticed that when peer review 
managers volunteer to manage they tend to favour ones with BoostBook 
docs. I think they also think it's another problem they don't have to 
think about during managing.

Niall

-- 
ned Productions Limited Consulting
http://www.nedproductions.biz/ 
http://ie.linkedin.com/in/nialldouglas/