Boost logo

Boost :

Subject: Re: [boost] [afio] AFIO review postponed till Monday
From: Niall Douglas (s_sourceforge_at_[hidden])
Date: 2015-07-22 12:56:17

On 21 Jul 2015 at 23:31, Glen Fernandes wrote:

> > I might be mistaken, but are you using undocumented NT APIs for the Windows specific implementation?
> The reason I asked the question was because, when glancing your
> repository, I saw comments like: "From

Ah, sorry. That site is basically a reference site for WINE and
ReactOS programmers. It's essentially Microsoft's kernel docs, but
with useful notes from the coal face as it were.

> You wrote:
> > All the kernel APIs it uses are from the official Microsoft kernel documentation
> Ah; this changes things. If you're only using documented APIs, that's
> not alarming.

After having slept on it, I lied to you earlier. There is exactly one
place I use an undocumented behaviour and it is here at
io/v2/detail/impl/afio_iocp.ipp#L370 where I use NtQueryObject with
the deprecated query ObjectNameInformation.

ObjectNameInformation was removed from the official docs if I
remember around NT 3.5 because it hangs when fed certain types of
handle (none of which AFIO would ever use), but it still works right
up to Windows 10. The reason I use it is because the alternatives are
all vastly slower and path() is on a performance critical path, and
if it ever were to change in the future the unit tests and all APIs
would all spectacularly fail so this is not something which would be

According to the ReactOS source code, ObjectNameInformation is used
very extensively throughout Microsoft code, including Explorer and a
raft of Office code. I judged it worth the risk for the gains.

> The bug you've hit and the response you received, though, might be an
> indication that the internal APIs aren't terrific to use in your
> portable library either. They're certainly acceptable to use in any
> system software or driver that is tested thoroughly for, and ships
> only for, specific Windows versions with less concern about binary
> forward compatibility.

I am unaware of any of the NT kernel APIs used by AFIO having ever
changed at all since their introduction into the kernel, most of
which happened in the early 1990s. Of course just because they have
never changed till now doesn't preclude their changing in the future,
but it's rather similar to POSIX deciding that the open() function
needs to be changed - they're going to add an open2() function
instead of changing the spec of the open() function in a breaking
way. Too much code hangs on the current NT kernel API that it is ever
worth changing existing APIs.

> I guess the test here is: How many potential AFIO users would have
> concerns about consuming AFIO in some cross-platform project when they
> find it uses internal APIs? (I'm would imagine most would refuse to
> use it outright if it were using any undocumented APIs). If the answer
> is "little to none", then you're probably fine. If not, it will
> probably work against you in the review.

I think this is the wrong question to ask.

The right question to ask is: do I mind using a Boost library which
skips the win32 layer to use the publicly documented kernel API
instead in order to gain:

* Much improved POSIX semantics including inodes, deleting and
renaming open files, POSIX timestamping, POSIX volume mount points,
no longer random fails when deleting directory trees, POSIX symlinks,
POSIX stat_t, POSIX statfs_t etc.

* Async everything, including metadata fetches, directory enumeration
and file byte range locking

* No win32 induced weirdnesses with paths (e.g. char limits, magic
character conversion, weird substring expansions)

* Up to 50% more performance for any operation consuming a path

* 10m item directories are not a problem, same as Linux and BSD

* Filesystem race freedom and explicitly documented race guarantees
per API

* Inode instead path item lookups

In exchange for these losses:

* No DOS shortname compatibility

* 10m item directories hang Explorer and the command line

* Generated win32 paths from NT kernel paths can confuse users as
they don't "look right"

If you have ever been surprised by a bug report where your filesystem
related code on Windows did not behave as you expected from POSIX, I
think AFIO is worth examining.

BTW while we're on the topic AFIO also directly uses the syscall
interface on Linux, BSD and OS X for directory enumeration and skips
the C library. This allows AFIO to handle > 10m item directories with
ease, plus it gives snapshot race freedom with the same guarantees as
on Windows.


ned Productions Limited Consulting

Boost list run by bdawes at, gregod at, cpdaniel at, john at