Re: [Filesystem] Directory entry caching

From: Chris Glover (c.d.glover_at_[hidden])
Date: 2020-05-13 03:32:02

On Sun, May 10, 2020 at 6:02 PM Niall Douglas <s_sourceforge_at_[hidden]>
wrote:

>
> So this turned into a weekend long thing to figure out, and I thank you
> for it, as we have a whole bunch of code here which assumes things which
> it turns out are out of date on recent Windows. FYI I had to break out
> the Windows Performance Recorder to figure out what the hell was going
> on in the Windows kernel, more on that shortly.
>
> Quick results for enumerating all of C:\ on my machine, warm cache:
>
> ```
> G:\boostish\cdglove>x64\Release\llfio.exe
> test-llfio found 1485232 files.
>  13.872090s wall, 1.734375s user + 12.109375s system = 13.843750s CPU
> (99.8%)
>
> G:\boostish\cdglove>x64\Release\win32.exe
> test-win32 found 1491222 files.
>  14.713440s wall, 2.328125s user + 12.375000s system = 14.703125s CPU
> (99.9%)
>
> G:\boostish\cdglove>x64\Release\boost.exe
> Exception thrown: filesystem::recursive_directory_iterator increment
> error: The system cannot find the path specified
> Up until fatal error test-boost_filesystem found 868657 files.
>  66.506680s wall, 5.187500s user + 61.234375s system = 66.421875s CPU
> (99.9%)
>
> G:\boostish\cdglove>x64\Release\std.exe
> Exception thrown: recursive_directory_iterator::operator++: The system
> cannot find the path specified.
> Up until fatal error test-std_filesystem found 870341 files.
>  63.407547s wall, 2.921875s user + 60.375000s system = 63.296875s CPU
> (99.8%)
> ```
>

Can confirm I am seeing the same results as you now. After updating llfio,
performance improved massively such that I was within 2x of FindNextFile.
Then after making your recommended changes to how I was using llfio, they
roughly converged.

I'm not convinced that this could replace a high level directory iterator
though. llfio is easy to use incorrectly as evidenced by the annotations
you needed to add to my code. That's not a criticism of the design, only a
commentary that it's sufficiently low level to cause some grief if one
isn't very careful. I have a few minor improvement ideas on that which we
can discuss off list.

I'll likely play with the unfinished work to directory_iterator to see if
any improvements can be made.

I pushed the updated changes to https://github.com/cdglove/fs-test. I think
I also fixed the access error exception you were getting in the boost test.
(added pop_on_error to the directory options).

Thanks for taking the time to look into all of this!

-- chris