Re: [boost] [afio] AFIO review postponed till Monday

Subject: Re: [boost] [afio] AFIO review postponed till Monday
From: Niall Douglas (s_sourceforge_at_[hidden])
Date: 2015-07-22 12:56:17

On 21 Jul 2015 at 23:31, Glen Fernandes wrote:

> >  I might be mistaken, but are you using undocumented NT APIs for the Windows specific implementation?
> 
> The reason I asked the question was because, when glancing your
> repository, I saw comments like: "From
> http://undocumented.ntinternals.net".

Ah, sorry. That site is basically a reference site for WINE and 
ReactOS programmers. It's essentially Microsoft's kernel docs, but 
with useful notes from the coal face as it were.

> You wrote:
> >  All the kernel APIs it uses are from the official Microsoft kernel documentation
> 
> Ah; this changes things. If you're only using documented APIs, that's
> not alarming.

After having slept on it, I lied to you earlier. There is exactly one 
place I use an undocumented behaviour and it is here at 
https://github.com/BoostGSoC13/boost.afio/blob/master/include/boost/af
io/v2/detail/impl/afio_iocp.ipp#L370 where I use NtQueryObject with 
the deprecated query ObjectNameInformation.

ObjectNameInformation was removed from the official docs if I 
remember around NT 3.5 because it hangs when fed certain types of 
handle (none of which AFIO would ever use), but it still works right 
up to Windows 10. The reason I use it is because the alternatives are 
all vastly slower and path() is on a performance critical path, and 
if it ever were to change in the future the unit tests and all APIs 
would all spectacularly fail so this is not something which would be 
missed.

According to the ReactOS source code, ObjectNameInformation is used 
very extensively throughout Microsoft code, including Explorer and a 
raft of Office code. I judged it worth the risk for the gains.

> The bug you've hit and the response you received, though, might be an
> indication that the internal APIs aren't terrific to use in your
> portable library either. They're certainly acceptable to use in any
> system software or driver that is tested thoroughly for, and ships
> only for, specific Windows versions with less concern about binary
> forward compatibility.

I am unaware of any of the NT kernel APIs used by AFIO having ever 
changed at all since their introduction into the kernel, most of 
which happened in the early 1990s. Of course just because they have 
never changed till now doesn't preclude their changing in the future, 
but it's rather similar to POSIX deciding that the open() function 
needs to be changed - they're going to add an open2() function 
instead of changing the spec of the open() function in a breaking 
way. Too much code hangs on the current NT kernel API that it is ever 
worth changing existing APIs.

> I guess the test here is: How many potential AFIO users would have
> concerns about consuming AFIO in some cross-platform project when they
> find it uses internal APIs? (I'm would imagine most would refuse to
> use it outright if it were using any undocumented APIs). If the answer
> is "little to none", then you're probably fine. If not, it will
> probably work against you in the review.

I think this is the wrong question to ask.

The right question to ask is: do I mind using a Boost library which 
skips the win32 layer to use the publicly documented kernel API 
instead in order to gain:

* Much improved POSIX semantics including inodes, deleting and 
renaming open files, POSIX timestamping, POSIX volume mount points, 
no longer random fails when deleting directory trees, POSIX symlinks, 
POSIX stat_t, POSIX statfs_t etc.

* Async everything, including metadata fetches, directory enumeration 
and file byte range locking

* No win32 induced weirdnesses with paths (e.g. char limits, magic 
character conversion, weird substring expansions)

* Up to 50% more performance for any operation consuming a path

* 10m item directories are not a problem, same as Linux and BSD

* Filesystem race freedom and explicitly documented race guarantees 
per API

* Inode instead path item lookups

In exchange for these losses:

* No DOS shortname compatibility

* 10m item directories hang Explorer and the command line

* Generated win32 paths from NT kernel paths can confuse users as 
they don't "look right"

If you have ever been surprised by a bug report where your filesystem 
related code on Windows did not behave as you expected from POSIX, I 
think AFIO is worth examining.

BTW while we're on the topic AFIO also directly uses the syscall 
interface on Linux, BSD and OS X for directory enumeration and skips 
the C library. This allows AFIO to handle > 10m item directories with 
ease, plus it gives snapshot race freedom with the same guarantees as 
on Windows.

Niall

-- 
ned Productions Limited Consulting
http://www.nedproductions.biz/ 
http://ie.linkedin.com/in/nialldouglas/