Boost logo

Boost :

Subject: Re: [boost] [AFIO] Formal review
From: Niall Douglas (s_sourceforge_at_[hidden])
Date: 2015-09-02 15:43:39

On 2 Sep 2015 at 7:20, Hartmut Kaiser wrote:

> > Limiting the discussion to just the problem of race free filesystem
> > on POSIX, imagine the problem of opening the sibling of a file in a
> > directory whose path is constantly changing. POSIX does not provide
> > an API for opening the parent directory of an open file descriptor.
> > To work around this, you must first get the canonical current path of
> > the open file descriptor, strip off the end, open that directory and
> > then use that as a base directory for opening a file with the same
> > leafname as your original file. You then loop all that if you fail to
> > open that leafname, or the leafname opened has a different inode.
> > Once you have the correct parent directory, you can open the sibling.
> > This is an example of where caching the stat_t of handle during open
> > saves syscalls and branches in more performance important APIs later
> > on.
> Sorry for asking a question again (and I'm sure Niall will ignore it because
> he judges it as 'disrespectful, petty, or juvenile', but perhaps somebody
> else can answer this):

You could treat me with the respect due to someone working in this
field for several years the same as the respect I treat you for your
ample experience in your domain. If I treated you the way you treat
me, you would explode as we have seen before with you shouting at
other people on this list who you felt were being disrespectful to

I am not incompetent. If I chose a design, it was for a reason. It
may have been a suboptimal choice, but it is not due to lack of
experience nor a lack of prior research. I have been programming for
over twenty years, and more than fifteen commercially. I have a very
different style and approach to you, but that does not make it
inferior or to be mocked or attacked just because I am imprecise in
your opinion.

> In another mail you said AFIO wouldn't cache anything, thus must do a lot of
> syscalls over and over again, which is the reason for it to be slow.

We don't second guess the OS. If the OS lies to us - and it does,
frequently - we pass through those lies exactly. Similarly, if you
loop requesting the length of a file, you'll get back exactly what
the OS tells us each time. We always ask the OS for the length if you
ask AFIO for the length. No caching.

However the semantics emulation and workaround code still needs to
correctly emulate any missing semantics on some platform in order to
provide a universal file system model to the AFIO user. AFIO provides
race-free sibling lookup which works just fine on Windows, but
requires some extra work on POSIX, so on POSIX it runs the algorithm
described above.

The difference is not unlike the difference between FreeBSD and Linux
with respect to POSIX. POSIX defines a universal portable standard,
and both Linux and FreeBSD implement most of that standard. FreeBSD
implements POSIX far more accurately than Linux, and has
correspondingly lower peak performance in everything from mutexs to
opening files.

However, once one includes the cost of the workarounds for Linux's
divergences from POSIX in application code, much of that performance
gap is eliminated. FreeBSD also scales out much better than Linux
because the lower peak performance in a 1-4 CPU machine is really a
situation with a higher fixed cost but a less steep slope on the
variable cost curve, so as you ramp up the hardware FreeBSD scales

AFIO is similar to POSIX here. It provides a universal standard
model. All platforms require workarounds to match that model, but the
workarounds vary between even Linux and FreeBSD. If AFIO must execute
extra syscalls on some particular platform in some particular case,
you should accept that as a sunk cost on that platform and not worth
worrying about in the bigger picture as the simplified application
code with fewer workarounds is likely to perform better, be easier to
maintain, and transparently improves as OS facilities improve. Your
application code is also much more portable.


ned Productions Limited Consulting

Boost list run by bdawes at, gregod at, cpdaniel at, john at