[boost] Re: Logging library - your own scenario!

From: Darryl Green (darryl.green_at_[hidden])
Date: 2005-05-19 22:59:02

> Mark Blewett <boost <at> blewett.nildram.co.uk> writes:

Thanks for writing my request for me :) but please see additional requirements
at the end.

> 
> 1) Each trace (log) many have zero or more observers (appender) with 
> independent control of what is seen.
> 
> For example our production systems run with an observer which only writes 
> only critical/error messages to a local file. However we also provide 
> off-site support, where we remotely connect (possibly multiple people 
> connecting) via tcp/ip to the logging system, where each person has their 
> own view and only sees the traces that they have enabled.

Me too (near enough).

> 
> 2) Redirecting cout / cerr to a trace.

Not on my list, but nice to have.

> 3) Different formats (modifiiers) for different observers.

Yes. This one is annoying because it turns the modifiers/appenders list into a
tree. I think this is where it becomes better to resort to a structured log
message format, where modifiers are only used to add additional structured data
such as timestamps to log messages, and where formatting this data is left up
to the appender. So long as it is easy to compose an appender from a "dumb"
writer that just receives a string (or that is just an ostream? but that gets
ugly in that ostreams do formatting...) plus a formatter taking the structured
log message I think this is nicer all round - users/support staff don't get:

17:56:02.123 ERROR: PID:1234 TID:4567 foobar.cpp:321 unhandled
some@$%^mangled_and_even_demangled_still_incomprehensible_typename event

and do get

17:56:02 ERROR: Unexpected event. Call Fred NOW. He knows where the nasty
detailed log is and how to read it.

(well ok, maybe not quite that descriptive, but you get the idea)

Because the actual appender is constructed like this (very rough - don't take
the exact method too seriously):

string user_log_formatter(log_msg m);
string debug_log_formatter(log_msg m);

logfile_appender user_log("/var/log/myapp.log", user_log_formatter);
logfile_appender debug_log("/var/log/mayapp/debug.log", debug_log_formatter);

> 
> For example for a file format we might like each line as "<date> <time> : 
> <trace> : <level> : <message>", where as we might also like to send the 
> same data across the network in xml / asn1 format, or even insert a row 
> into a database table.

I see the case where the same message data is presented in different formats
(eg ASN.1 vs text log entry) as being simply a marshalling format issue
(assuming that the log message is structured) which is handled similarly to the
above. Actually I have considered just marshalling the log message into a
string as ASN.1 BER and having the formatters pull that apart if/as required.

I do have some extra requirements/constraints:

a) No singletons. That is, I want to be able to have logging using multiple
managers with different policies in a single app. While this may sound a little
weird, I need something that looks an awful lot like logging to actually log
application level events, in specific formats etc independent of application
debug/error logging. This is a failing of the logging lib I had been using.

b) Dynamic/scoped logs are essential. I know they are in the design already,
but it is one feature that doesn't seem to get many requests and, despite (e)
below, one feature I can't do without.

c) There are different interpretations of the named logs/keywords/tags/channels
being kicked around. I'd just like to register a definite vote for the named
log approach as currently implemented. It achieves everything I've ever wanted
from this sort of attribute, very efficiently. If some other post filtering
naming is needed (it isn't by me), it can just be another attribute in a
structured log message.

d) Thread safety is required (as is the ability to turn it off). Minimise lock
scope and hold time. Appenders must implement their own thread safety anyway
(unless a global lock is used, which would be bad) given that they can be
shared by multiple logs. Except during updates to settings the logs themselves
should be reenterant.

e) If feature lists lead to code size/data size/performance compromises thats
fine, so long as I can turn off features I don't want. Layering or policies
please. Embedded systems may have a lot more memory and MIPS than they once
did, but it's a long way from infinite, and an order of magnitude or more below
desktops (ymmmv). I don't expect the logging lib to be tailored to this
environment (which is hard to define in a general way anyhow) but
configurability is key. Logging is not the deliverable - it must be
efficient otherwise it will just get left out.

Darryl.