From: Chris Glover (c.d.glover_at_[hidden])
Date: 2020-05-13 03:32:02
On Sun, May 10, 2020 at 6:02 PM Niall Douglas <s_sourceforge_at_[hidden]>
> So this turned into a weekend long thing to figure out, and I thank you
> for it, as we have a whole bunch of code here which assumes things which
> it turns out are out of date on recent Windows. FYI I had to break out
> the Windows Performance Recorder to figure out what the hell was going
> on in the Windows kernel, more on that shortly.
> Quick results for enumerating all of C:\ on my machine, warm cache:
> test-llfio found 1485232 files.
> 13.872090s wall, 1.734375s user + 12.109375s system = 13.843750s CPU
> test-win32 found 1491222 files.
> 14.713440s wall, 2.328125s user + 12.375000s system = 14.703125s CPU
> Exception thrown: filesystem::recursive_directory_iterator increment
> error: The system cannot find the path specified
> Up until fatal error test-boost_filesystem found 868657 files.
> 66.506680s wall, 5.187500s user + 61.234375s system = 66.421875s CPU
> Exception thrown: recursive_directory_iterator::operator++: The system
> cannot find the path specified.
> Up until fatal error test-std_filesystem found 870341 files.
> 63.407547s wall, 2.921875s user + 60.375000s system = 63.296875s CPU
Can confirm I am seeing the same results as you now. After updating llfio,
performance improved massively such that I was within 2x of FindNextFile.
Then after making your recommended changes to how I was using llfio, they
I'm not convinced that this could replace a high level directory iterator
though. llfio is easy to use incorrectly as evidenced by the annotations
you needed to add to my code. That's not a criticism of the design, only a
commentary that it's sufficiently low level to cause some grief if one
isn't very careful. I have a few minor improvement ideas on that which we
can discuss off list.
I'll likely play with the unfinished work to directory_iterator to see if
any improvements can be made.
I pushed the updated changes to https://github.com/cdglove/fs-test. I think
I also fixed the access error exception you were getting in the boost test.
(added pop_on_error to the directory options).
Thanks for taking the time to look into all of this!
Boost list run by bdawes at acm.org, gregod at cs.rpi.edu, cpdaniel at pacbell.net, john at johnmaddock.co.uk