Re: boost.log's text_multifile_backend is not performant on windows - perhaps it should be documented?

From: Nicholas Neumann (nick2002_at_[hidden])
Date: 2021-03-04 21:40:42

Peter Dimov via Boost wrote:
> The right thing to do would probably be to implement a deferred close
where a
> file lingers around for a second or so, then is closed by a background
thread. If
> it's used again within this second, it doesn't need to be closed and
reopened.

I'd also be happy with just an option on the backend that requires
opting-in to use an unbounded cache. It would be off by default but could
be turned on just like flushing on the regular file stream backend. Like
Andrey pointed out, if your use case involves an unbounded number of log
files, using a cache is going to have complications and shortfalls. But if
your use case is like mine - using the multifile backend instead of having
to declare a bunch of regular streams and filters - just keeping all of the
files open until program termination is exactly what you want.

Thinking a little bit differently, the multifile backend could allow for a
client-supplied callback, run after each record consume, which tells the
backend whether to close the file or leave it open in an unbounded cache.
Then cache management is completely customizable but also not the backend's
problem. This probably makes a lot of sense for the different use cases too:

- By default, there is no callback. The file is always closed.
- For my use case, the callback always returns don't close the file.
- For the use case with a log file for each connection/business process
entity/etc, the client probably knows when they are done with the log file
for that entity (e.g., the socket is closed). They are then responsible for
the callback saying it is time to close the file when the last log entry
for it is issued
- For some unimagined use case, the client can customize however they want.
The callback could even provide info about the open files, last time used,
etc, if the client wanted to do something super customized.

Thoughts?

On Thu, Mar 4, 2021 at 1:56 PM Peter Dimov via Boost <boost_at_[hidden]>
wrote:

> Andrey Semashev wrote:
> > On 3/4/21 10:04 PM, Peter Dimov via Boost wrote:
> > > Andrey Semashev wrote:
> > >
> > >> Unfortunately, text_multifile_backend is supposed to open and close
> > >> file on every log record, as the file name is generated from the log
> record.
> > >
> > > If you maintain a small LRU cache, will it help?
> >
> > That's what I had in mind, but it will have its own problems.
> >
> > One of the main use cases for text_multifile_backend is when you want to
> > maintain a separate log for each of the user's business process entity
> (e.g. a
> > session or a connection of some sort). There can be any number of such
> > entities, and I can't come up with a reasonable limit for a cache in the
> sink
> > backend. Naturally, the cache will be ineffective if the number of
> actively
> > logging entities exceeds its capacity. At the same time I don't want to
> keep any
> > stale open files in the cache.
> >
> > It seems, I'll have to maintain some sort of a timeout for closing files
> and
> > cleaning up the cache. But I will only be able to perform the cleanup on
> log
> > records, which means I'll still have stale open files in cache if
> there's no logging
> > happening.
> >
> > Keeping open files is not only a concern from resource usage
> perspective. It
> > may affect user's experience, as e.g. on Windows he won't be able to
> remove
> > the open files or the containing directory.
> > Deleting an open file on POSIX systems will succeed, but it will also
> mean some
> > subsequent log records can be lost as they will be written to the
> deleted file
> > instead of the new one.
>
> It's definitely not trivial.
>
> The right thing to do would probably be to implement a deferred close
> where a
> file lingers around for a second or so, then is closed by a background
> thread. If
> it's used again within this second, it doesn't need to be closed and
> reopened.
>
>
> _______________________________________________
> Unsubscribe & other changes:
> http://lists.boost.org/mailman/listinfo.cgi/boost
>