[boost] Re: Re: boost::filesystem - directory_iteratormakesoptimizationhard

From: Beman Dawes (bdawes_at_[hidden])
Date: 2005-06-02 09:02:09

"Peter Dimov" <pdimov_at_[hidden]> wrote in message 
news:000901c56770$deb96fe0$6401a8c0_at_pdimov2...
> Beman Dawes wrote:
>
>> * That is a lot of additional interface complexity to support an
>> optimization that applies to Windows but not POSIX. Some of the other
>> schemes (which involved additional overloads to specific operations
>> functions) had less visible impact on the interface.
>>
>> * There have been no timings to indicate the inefficiency of the
>> current interface actually impacts production applications.
>>
>> * AFAICS, there is nothing in the current interface which prevents
>> caching of directory entries. Caching would probably aid more use cases 
>> than
>> proposed changes to interfaces. But both caching and user dirent
>> storage introduce serious additional race conditions. Not a good
>> thing. In fact a showstopper unless cache management is introduced,
>> further complicating the interface.
>
> As a data point, PHP does stat caching:
>
> http://www.php.net/manual/en/function.is-dir.php

Interesting. Thanks for the reference.

> because of performance issues. Stat caching does apply to POSIX systems as 
> well. Either a transparent caching scheme or is_* overloads taking a 
> directory iterator may work.

Yes, and one or the other of those would be my preferred solution if we ever 
decide to do something about the issue.

> The race conditions argument isn't very strong. The information returned 
> by is_directory is inherently race-prone, caching or no caching. I won't 
> be surprised if an OS does its own per-process stat caching, for instance.

I'd be very surprised if modern OS implementations didn't do some caching, 
at least at the disk page level, if not also at the directory or dirent 
level. But the OS is more likely than a user program to know when to refresh 
or flush a cache entry. The belief that there is probably OS caching is one 
of the reasons I'd like to see actual performance measurements in a 
realistic use case before worrying about efficiency.

--Beman