Re: boost.log's text_multifile_backend is not performant on windows - perhaps it should be documented?

From: Nicholas Neumann (nick2002_at_[hidden])
Date: 2021-03-05 19:58:53

After some more playing with the backend with an unbounded cache, I had a
couple interesting findings to share.

First, by optimizing stuff on the client side, I was able to get to about
67K log entries per second, with flush after each consume.

I profiled this (both in debug and release) and noticed a lot of time spent
in path hasing/key compares (cache is just a std::unordered_map from
boost::filesystem::path to std::unique_ptr<boost::filesystem::ofstream>).
Changing the cache to be indexed by string (and only forming a
boost::filesystem::path when a cached stream doesn't exist) improved
throughput to 100K log entries per second, which is pretty substantial.

All of that is to say that for my use case too I now see a nice reason to
have more than just a boolean callback. I'm not sure that means I'd want or
need ability to do the file opening and closing myself, but in some sense,
why not? The backend could provide the default functions for the callback
to call, making it easy for those that don't want to customize, but
customizable for those that do.

It also means that rather than having the callback have the path as an
input, I'd want it to just give me the log record. Forming the path and
working with it can be a lot more expensive than looking at the single
entry in the log record that determines the path. I realize now I've gone
from "a callback mechanism would be great to make this backend performant
on windows" to "it would be great if the callback mechanism allowed me all
kinds of options to squeeze even more performance out of the backend". :-)
But maybe that starts to make it attractive to linux users as well?