Re: boost.log's text_multifile_backend is not performant on windows - perhaps it should be documented?

From: Niall Douglas (s_sourceforge_at_[hidden])
Date: 2021-03-05 12:56:26

On 05/03/2021 10:45, Andrey Semashev via Boost wrote:

>> On Windows, files have an intermediate state of being-deleted. So what
>> you can do is:
>>
>> 1. Have an async thread keep a pool of file handles full. You open
>> these handles to randomly named files, and once opened, you mark them
>> for deletion.
>>
>> 2. When you need to "create" a new file, you unmark a handle for
>> deletion, and rename it to what you need. All this can be done with
>> SetFileInformationByHandle.
>>
>> 3. If there is sudden process exit, or sudden power loss, all files
>> marked for deletion are actually deleted. So no need to clean caches
>> on a timer or anything like that.
>>
>> If you are on a new enough Windows 10, you can make the entire file
>> entry vanish when you mark it for deletion, rather like O_TMPFILE on
>> Linux. You can then make it reappear on an as-needed basis. On older
>> Windows, the file entry will still appear in directory listings, but
>> will be unopenable.
> 
> An interesting idea, but I'd prefer if the backend behaved roughly the
> same on POSIX and Windows. Besides, I'd prefer to avoid spawning threads
> in the backend, as Boost.Log currently only creates threads for
> asynchronous logging and never else. Even then the user can provide his
> own threads.

I appreciate that, but the two systems are very different. Using the
same design patterns for both, imposing one design pattern on a platform
on which it performs very badly, personally speaking I don't find that a
high Quality of Implementation.

i.e. When in Rome, do as the Romans do.

>>> Keeping open files is not only a concern from resource usage
>>> perspective. It may affect user's experience, as e.g. on Windows he
>>> won't be able to remove the open files or the containing directory. 
>>
>> You can delete or rename open files on Windows just fine. It's just
>> std::ofstream opens files in a way where you can't.
> 
> But the containing directory remains locked, doesn't it?

Locked in the sense that it cannot be renamed or deleted, yes.

The simple workaround is to rename the file out of its directory before
you delete it, then the containing directory behaves like on POSIX.

Recent Windows 10 implement "POSIX delete" by renaming the file into a
magic hidden NTFS folder in the root called \$Extend\$Deleted, and
marking it as in-process-of-being-deleted. That same technique works
right back to NT 3.5.

>> Another suggestion is that any decent Windows AV will not scan files
>> incapable of holding executable code for binary viruses. std::ofstream
>> opens files capable of holding executable code, so everything written
>> to them must be scanned deeply. If you instead create a file without
>> executability privileges, the AV should only scan for a small subset
>> of potential malcontent.
> 
> This might be useful to do. How do you specify that a file is not
> executable on Windows? There are dwDesiredAccess flags for
> CreateFile[1], but my understanding is that they describe the access
> granted through the returned handle and not permissions of the actual file.
> 
> [1]:
> https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-createfilea

A file can be capable of execution according to its ACL.

A file handle can be capable of executing code according to its
privileges (dwDesiredAccess contains FILE_GENERIC_EXECUTE).

You can create a file with an ACL not permitting execution using
lpSecurityAttributes, or use hTemplateFile to clone a security
descriptor. Or, you can temporarily adjust the current thread's
impersonation token such that specifying NULL to CreateFile means files
created by that thread aren't executable. The former is best when you
can adjust source code, the latter is best when you are calling third
party code and you want it to create different permissioned files.

Note that I don't actually know if the Windows default AV is clever
enough to not deep scan files without Execute ability. I do know it
specially skips files signed by a whitelist of Microsoft approved
signing authorities, and it was written by Microsoft themselves who
suffer the most from it, so your chances are good.

(Incidentally, to anyone not aware yet, you can exclude whole directory
trees from Microsoft Defender realtime analysis. Just add your MSVC
directory, your WSL directory (this makes a HUGE difference to WSL), all
your build directories, all tooling and source directories, and so on.
Build times approximately halve)

Niall