Re: boost.log's text_multifile_backend is not performant on windows - perhaps it should be documented?

From: Niall Douglas (s_sourceforge_at_[hidden])
Date: 2021-03-05 09:36:13

On 04/03/2021 19:34, Andrey Semashev via Boost wrote:

>>> Unfortunately, text_multifile_backend is supposed to open and close 
>>> file on
>>> every log record, as the file name is generated from the log record.
>>
>> If you maintain a small LRU cache, will it help?
> 
> That's what I had in mind, but it will have its own problems.
> 
> One of the main use cases for text_multifile_backend is when you want to 
> maintain a separate log for each of the user's business process entity 
> (e.g. a session or a connection of some sort). There can be any number 
> of such entities, and I can't come up with a reasonable limit for a 
> cache in the sink backend. Naturally, the cache will be ineffective if 
> the number of actively logging entities exceeds its capacity. At the 
> same time I don't want to keep any stale open files in the cache.
> 
> It seems, I'll have to maintain some sort of a timeout for closing files 
> and cleaning up the cache. But I will only be able to perform the 
> cleanup on log records, which means I'll still have stale open files in 
> cache if there's no logging happening.

You're thinking in terms of POSIX. On Windows, files have an 
intermediate state of being-deleted. So what you can do is:

1. Have an async thread keep a pool of file handles full. You open these 
handles to randomly named files, and once opened, you mark them for 
deletion.

2. When you need to "create" a new file, you unmark a handle for 
deletion, and rename it to what you need. All this can be done with 
SetFileInformationByHandle.

3. If there is sudden process exit, or sudden power loss, all files 
marked for deletion are actually deleted. So no need to clean caches on 
a timer or anything like that.

If you are on a new enough Windows 10, you can make the entire file 
entry vanish when you mark it for deletion, rather like O_TMPFILE on 
Linux. You can then make it reappear on an as-needed basis. On older 
Windows, the file entry will still appear in directory listings, but 
will be unopenable.

> Keeping open files is not only a concern from resource usage 
> perspective. It may affect user's experience, as e.g. on Windows he 
> won't be able to remove the open files or the containing directory. 

You can delete or rename open files on Windows just fine. It's just 
std::ofstream opens files in a way where you can't.

Another suggestion is that any decent Windows AV will not scan files 
incapable of holding executable code for binary viruses. std::ofstream 
opens files capable of holding executable code, so everything written to 
them must be scanned deeply. If you instead create a file without 
executability privileges, the AV should only scan for a small subset of 
potential malcontent.

Niall