Re: [boost] [afio] Formal review of Boost.AFIO

Subject: Re: [boost] [afio] Formal review of Boost.AFIO
From: Andreas Schäfer (gentryx_at_[hidden])
Date: 2015-08-31 15:14:36

On 18:19 Mon 31 Aug     , Niall Douglas wrote:
> On 31 Aug 2015 at 17:20, Andreas Schäfer wrote:
> 
> > - The naive OpenMP example is using a thread pool, AFIO is using these
> >   elaborate, efficient futures, continuations etc. I'd have expected
> >   this to be more efficient, especially given the nature of the OpenMP
> >   example (which does a sequential directory walk before spawning
> >   threads).
> 
> More code executed usually means slower performance.

This is a vague answer and doesn't give me any insight WRT the problem
at hand.

> > - Even if we assume that AFIO can't be faster than an OpenMP code in
> >   this setting: why is the overhead so high? What is making it >2x
> >   slower?
> 
> I have been as transparent and honest about the inefficiencies in
> AFIO as I can be. I have listed copious benchmarks and graphs in the
> AFIO documentation which show AFIO in as poor a light as I can make
> it. I have identified another round of low hanging fruit which I feel
> lightweight futures will help me solve, and I am about to go
> implement. What more do you want here?

I want an explanation. Saying that were open or that more code was
being executed is not particularly enlightening. Since you wrote that
both examples essentially execute the same file system calls. I'd like
to know which functions within AFIO burn those extra cycles. If it was
my code I'd do a profiling run to assess exactly this.

Actually I don't want to know this for myself, but I believe it would
be valuable to you to steer your future development activities.

> In the end AFIO does a lot more than a naïve OpenMP solution, and I
> never expect it to be quicker without a true kernel async backend to
> offset the extra work performed. It executes more syscalls and does
> more processing. In exchange, you get much superior guarantees and
> reliability.

Why syscalls are executed in addition? I was assuming both codes would
run the same file system syscalls.

> If you don't want those extra guarantees, then don't use AFIO. As I
> have suggested on multiple occasions now, ASIO has a perfectly fine
> async file i/o library.

So AFIO's unique selling point is now supposed to be only reliability
WRT race conditions? I'll try to remember this.

> > > What AFIO provides is a *portable* API for async file system and file
> > > i/o. It provides the least benefit on Linux of any of the platforms,
> > > but that is not AFIO's fault.
> >
> > Sorry, but that's not true. It remains AFIO's fault as long as you
> > can't explain where this slowdown comes from. Why don't you, say,
> > default to a naive, simple OpenMP implementation on Linux?
> 
> I have repeatedly explained where I believe slowdowns come from.

Your speculations, as far as I can remember, were: a) no good API in
Linux <4.0, b) more code being executed, c) more syscalls being
issued. Did I miss any other explanation?

I'm asking this because profiling is part of my job. I do it to find
out why my (or other people's) code is slow. The more I do it the less
I trust my own speculations. Measurements trump intuition.

> You keep choosing to not listen, twist and cherry pick, and harrass
> me again and again over the same items.

I ask the same questions from different angles to get answers.
Mistaking that for harassment is troubling.

> I could guess at your motives given you are also in German HPC, but
> I will leave it at that.

What is that supposed to mean?

> I have now had enough of this harrassment and repeatedly explaining
> the same thing over and over. You have your answers and you have made
> your vote. Please now be quiet and let others speak.

It's sad that you chose to view my questions as harassment while they
were really an opportunity for you to resolve bugs (the segfault on my
system) and demonstrate your technical excellence by lecturing me on
your design.

Kind regards
-Andreas

-- 
==========================================================
Andreas Schäfer
HPC and Grid Computing
Department of Computer Science 3
Friedrich-Alexander-Universität Erlangen-Nürnberg, Germany
[phone]
PGP/GPG key via keyserver
http://www.libgeodecomp.org
==========================================================
(\___/)
(+'.'+)
(")_(")
This is Bunny. Copy and paste Bunny into your
signature to help him gain world domination!