Boost logo

Boost :

Subject: Re: [boost] [GSoC] [Boost.Hana] Formal review request
From: Niall Douglas (s_sourceforge_at_[hidden])
Date: 2014-07-29 20:00:11


On 29 Jul 2014 at 17:02, Louis Dionne wrote:

> > Also, regarding formal review, I personally would feel uncomfortable
> > accepting a library that only works with a single version of clang. I
> > would feel much happier if you got trunk GCC working, even if that
> > means workarounds.
>
> That would mean a lot of workarounds.

Not necessarily. You could wait until GCC trunk catches up instead,
helping it along by adding however many bug reports as is necessary.
Timing well a formal review is often as important as the quality of
your library.

I always feel heebie jeebies about code which works only on one
compiler. For me to vote yes for a Boost library to enter Boost I
need to feel it is well tested and reliable and has had all its kinks
knocked out. I struggle to see me feeling this with a library which
can't be tested widely.

Instead of heading straight into the community review queue, perhaps
a few rounds of intermediate informal reviews like this one? I'd
particularly like to see Eric and Joel's opinion of your library so
far too.

> Basically, I got a benchmark for almost every operation of almost every
> sequence that's supported by Hana (including those adapted from external
> libraries), but I'm not sure yet of how to group them in the documentation
> (per operation? per sequence? per type class?).
>
> The problem is made worse by two things:
>
> - It only makes sense to benchmark components that are isomorphic. For
> example, what does it mean to benchmark a std::tuple against a
> mpl::vector? Not much, because the price you pay for std::tuple
> gives you the ability to hold values, whereas mpl::vector can only
> hold types. We don't want to compare apples with oranges, and the
> grouping of benchmarks should be influenced by that.

I guess it's like when choosing between list, map and vector from the
STL. All are "equivalent", but you need to know how each might scale
out for what you need. Most users won't just known automatically.

My problem always with MPL98 was I had no idea what was fast or slow
for my use cases.

> - How do we handle different compilers? Right now, all benchmarks are
> produced only on Clang, which is OK because it's the only compiler that
> can compile the library. When there is more than one compiler, how do we
> generate the benchmarks for all of them, and how do we integrate the
> benchmarks in the documentation?

You're right this is more of a name and shame thing between AST
compilers. MSVC is very different though, and you may eventually need
a separate line labelled "MSVC" just for it.

> > I'd also like to see unit testing that verified that the current
> > compiler being tested has a time and space benchmark curve matching
> > what is expected. It is too easy for code to slip in or the compilers
> > themselves to gain a bug which creates pathological metaprogramming
> > performance. Better to have Travis CI trap that for you than head
> > scratching and surprises later.
>
> I thought about doing this, but I did not because I thought it was a HUGE
> undertaking to automate it.

No, it's easier than you think. Have a look at
https://ci.nedprod.com/ whose default dashboard shows a graph
labelled "RUDP performance". This tracks performance of a build over
time to ensure performance doesn't regress. All you need is for your
performance test tool to output some CSV, a Jenkins Plot plugin does
the rest.

> I think we're better off integrating the
> benchmarks in the documentation and then when something is really weird,
> we just have to look at the generated benchmarks and see what's wrong.
> If someone can suggest a way to do it automatically that won't take me weeks
> to set up, I'm interested.

Mastering Jenkins takes months, but once mastered configuring all
sorts of test scenarios becomes trivial. I'd actively merge your
Jenkins/Travis output into your docs too, it is nowadays an online
world.

> Also, I wouldn't want to slip in the trap of testing the compiler; testing
> Hana is a large enough task as it is (341 tests + 165 examples as we speak).

I think testing modern C++ is always testing the compiler. It's why I
develop first on MSVC, I stand a good chance of porting that to GCC
or clang, less so the other way round.

> > Finally, there are ways and means for doxygen docs to automatically
> > convert into BoostBook docs. You'll need to investigate those before
> > starting a formal review. Tip: look into how Geometry/AFIO does the
> > doxygen conversion, it's brittle but it is easier than the others.
>
> Is it mandatory for a Boost library to have BoostBook documentation?
> I'd like to stay as mainstream as possible in the tools I use and reduce
> the number of steps in the build/documentation process for the sake of
> simplicity. Is there a gain in generating the documentation in BoostBook?

As you'll find when formal review comes, to pass isn't about how good
your library is, it's about eliminating as many rational objections
others can think of.

BoostBook format documentation is a very easy way of crossing
permanently off a raft of potential objections. Other categories of
objection will be far harder to address, trust me. I suspect the
hardest will probably be the "so what?" objection in all the guises
it manifests, as it always is for any library which is particularly
novel.

All that said, I have little love for the Boost documentation format,
I think it outdated, brittle, and inflexible, and it does a lousy job
of automating reference documentation. But it could be a lot worse,
and I have been forced in the past to use a lot worse.

Besides, you'll invest five days or so of wishing pain on those
responsible for the tools, and once it's working you'll never need
touch it again. I did find it took some months to find and fix all
the corner cases in the doc output though, and even now PDF
generation from AFIO's docs are a joke due to the long template
strings.

Anyway, it's up to you. BTW, I've noticed that when peer review
managers volunteer to manage they tend to favour ones with BoostBook
docs. I think they also think it's another problem they don't have to
think about during managing.

Niall

-- 
ned Productions Limited Consulting
http://www.nedproductions.biz/ 
http://ie.linkedin.com/in/nialldouglas/



Boost list run by bdawes at acm.org, gregod at cs.rpi.edu, cpdaniel at pacbell.net, john at johnmaddock.co.uk