Re: [boost] [AFIO] Formal review

Subject: Re: [boost] [AFIO] Formal review
From: Niall Douglas (s_sourceforge_at_[hidden])
Date: 2015-09-02 15:43:39

On 2 Sep 2015 at 7:20, Hartmut Kaiser wrote:

> > Limiting the discussion to just the problem of race free filesystem
> > on POSIX, imagine the problem of opening the sibling of a file in a
> > directory whose path is constantly changing. POSIX does not provide
> > an API for opening the parent directory of an open file descriptor.
> > To work around this, you must first get the canonical current path of
> > the open file descriptor, strip off the end, open that directory and
> > then use that as a base directory for opening a file with the same
> > leafname as your original file. You then loop all that if you fail to
> > open that leafname, or the leafname opened has a different inode.
> > Once you have the correct parent directory, you can open the sibling.
> > This is an example of where caching the stat_t of handle during open
> > saves syscalls and branches in more performance important APIs later
> > on.
> 
> Sorry for asking a question again (and I'm sure Niall will ignore it because
> he judges it as 'disrespectful, petty, or juvenile', but perhaps somebody
> else can answer this): 

You could treat me with the respect due to someone working in this 
field for several years the same as the respect I treat you for your 
ample experience in your domain. If I treated you the way you treat 
me, you would explode as we have seen before with you shouting at 
other people on this list who you felt were being disrespectful to 
you.

I am not incompetent. If I chose a design, it was for a reason. It 
may have been a suboptimal choice, but it is not due to lack of 
experience nor a lack of prior research. I have been programming for 
over twenty years, and more than fifteen commercially. I have a very 
different style and approach to you, but that does not make it 
inferior or to be mocked or attacked just because I am imprecise in 
your opinion.

> In another mail you said AFIO wouldn't cache anything, thus must do a lot of
> syscalls over and over again, which is the reason for it to be slow. 

We don't second guess the OS. If the OS lies to us - and it does, 
frequently - we pass through those lies exactly. Similarly, if you 
loop requesting the length of a file, you'll get back exactly what 
the OS tells us each time. We always ask the OS for the length if you 
ask AFIO for the length. No caching.

However the semantics emulation and workaround code still needs to 
correctly emulate any missing semantics on some platform in order to 
provide a universal file system model to the AFIO user. AFIO provides 
race-free sibling lookup which works just fine on Windows, but 
requires some extra work on POSIX, so on POSIX it runs the algorithm 
described above.

The difference is not unlike the difference between FreeBSD and Linux 
with respect to POSIX. POSIX defines a universal portable standard, 
and both Linux and FreeBSD implement most of that standard. FreeBSD 
implements POSIX far more accurately than Linux, and has 
correspondingly lower peak performance in everything from mutexs to 
opening files.

However, once one includes the cost of the workarounds for Linux's 
divergences from POSIX in application code, much of that performance 
gap is eliminated. FreeBSD also scales out much better than Linux 
because the lower peak performance in a 1-4 CPU machine is really a 
situation with a higher fixed cost but a less steep slope on the 
variable cost curve, so as you ramp up the hardware FreeBSD scales 
better.

AFIO is similar to POSIX here. It provides a universal standard 
model. All platforms require workarounds to match that model, but the 
workarounds vary between even Linux and FreeBSD. If AFIO must execute 
extra syscalls on some particular platform in some particular case, 
you should accept that as a sunk cost on that platform and not worth 
worrying about in the bigger picture as the simplified application 
code with fewer workarounds is likely to perform better, be easier to 
maintain, and transparently improves as OS facilities improve. Your 
application code is also much more portable.

Niall

-- 
ned Productions Limited Consulting
http://www.nedproductions.biz/ 
http://ie.linkedin.com/in/nialldouglas/