Re: boost.log's text_multifile_backend is not performant on windows - perhaps it should be documented?

From: Andrey Semashev (andrey.semashev_at_[hidden])
Date: 2021-03-05 10:45:17

On 3/5/21 12:36 PM, Niall Douglas via Boost wrote:
> On 04/03/2021 19:34, Andrey Semashev via Boost wrote:
> 
>>>> Unfortunately, text_multifile_backend is supposed to open and close 
>>>> file on
>>>> every log record, as the file name is generated from the log record.
>>>
>>> If you maintain a small LRU cache, will it help?
>>
>> That's what I had in mind, but it will have its own problems.
>>
>> One of the main use cases for text_multifile_backend is when you want 
>> to maintain a separate log for each of the user's business process 
>> entity (e.g. a session or a connection of some sort). There can be any 
>> number of such entities, and I can't come up with a reasonable limit 
>> for a cache in the sink backend. Naturally, the cache will be 
>> ineffective if the number of actively logging entities exceeds its 
>> capacity. At the same time I don't want to keep any stale open files 
>> in the cache.
>>
>> It seems, I'll have to maintain some sort of a timeout for closing 
>> files and cleaning up the cache. But I will only be able to perform 
>> the cleanup on log records, which means I'll still have stale open 
>> files in cache if there's no logging happening.
> 
> You're thinking in terms of POSIX.

Indeed. I'm not using Windows very often, so I might be biased. Frankly, 
I'm writing code with POSIX in mind by default and add workarounds for 
Windows, when necessary.

> On Windows, files have an 
> intermediate state of being-deleted. So what you can do is:
> 
> 1. Have an async thread keep a pool of file handles full. You open these 
> handles to randomly named files, and once opened, you mark them for 
> deletion.
> 
> 2. When you need to "create" a new file, you unmark a handle for 
> deletion, and rename it to what you need. All this can be done with 
> SetFileInformationByHandle.
> 
> 3. If there is sudden process exit, or sudden power loss, all files 
> marked for deletion are actually deleted. So no need to clean caches on 
> a timer or anything like that.
> 
> If you are on a new enough Windows 10, you can make the entire file 
> entry vanish when you mark it for deletion, rather like O_TMPFILE on 
> Linux. You can then make it reappear on an as-needed basis. On older 
> Windows, the file entry will still appear in directory listings, but 
> will be unopenable.

An interesting idea, but I'd prefer if the backend behaved roughly the 
same on POSIX and Windows. Besides, I'd prefer to avoid spawning threads 
in the backend, as Boost.Log currently only creates threads for 
asynchronous logging and never else. Even then the user can provide his 
own threads.

>> Keeping open files is not only a concern from resource usage 
>> perspective. It may affect user's experience, as e.g. on Windows he 
>> won't be able to remove the open files or the containing directory. 
> 
> You can delete or rename open files on Windows just fine. It's just 
> std::ofstream opens files in a way where you can't.

But the containing directory remains locked, doesn't it?

> Another suggestion is that any decent Windows AV will not scan files 
> incapable of holding executable code for binary viruses. std::ofstream 
> opens files capable of holding executable code, so everything written to 
> them must be scanned deeply. If you instead create a file without 
> executability privileges, the AV should only scan for a small subset of 
> potential malcontent.

This might be useful to do. How do you specify that a file is not 
executable on Windows? There are dwDesiredAccess flags for 
CreateFile[1], but my understanding is that they describe the access 
granted through the returned handle and not permissions of the actual file.

[1]: 
https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-createfilea