Re: [boost] A meta-proposal for logging

From: Scott Woods (scottw_at_[hidden])
Date: 2007-04-10 19:12:36

Hi Gareth and others,

----- Original Message ----- 
From: "Gareth Buxton" <gareth.buxton_at_[hidden]>
To: <boost_at_[hidden]>
Sent: Monday, April 09, 2007 9:13 PM
Subject: Re: [boost] A meta-proposal for logging

>
>
> Hi Andrey,
>> You see, in your example the filter would get "maskable& line", not
>> "line_number& line" as the logging library doesn't know about
>> "line_number" class. This means that you will have to make some type
>> dispatch to get the real attribute type, and I'm afraid this will be
>> a performance killer.

What constitutes death? i.e. how do you know what number of additional
CPU cycles would change the mind of a potential adopter.

There is no real boundary. It is way too dependent on the circumstance.

>>
>>
> It could be but I don't think it necessarily has to. My solution would
> be to tightly couple the filter with the attributes that are to be
> filtered on the basis that if the user is adding the attributes then the
> use knows how best to filter based on them.

<snip>

> I tend to think that some amount of coupling is inevitable, for the most
> part I would imagine attributes would be well known, frequently used
> types like time (time_t), line_number (int) etc. and would be provided
> as 'standard' in the library.
>

I would have liked to respond to the individual points raised here but there
are a lot of them.

If there is something useful I can add it is most likely a brief summary of
a logging system recently completed. I balanced many of the issues
raised here.

* Logging is a stream of mixed-type records or discriminated variants.
* In application code, logging looks like;
    log_channel << "Reached here";
    log_channel << file_IO_problem;
    log_channel << server_snapshot();
    log_channel << stack_trace();
* The stream of objects is asynchronously queued, sent over a
    network connection, collated, time stamped and stored using a
    proprietary format and file management scheme.
* A dedicated viewer allows entry of arbitrary filtering expressions
   that typically results in a window-per record type, i.e. a fixed set
   of attributes.
* Users can navigate inside the filtered view by entering further
   search criteria or by clicking on an auto-generated time breakdown
   of the view.
* The record types are "user defined". Registration of a type includes
    enough information for the viewer to display full attribute information.
* The logging applications (i.e. the source of log records) communicate
   with the server using a formalized protocol. An obvious message (or
    signal) within this protocol is LOG (i.e. the connecting client is 
transferring
    a log record, an instance of a discriminated variant).
* Other less obvious protocol messages involve the distribution of the
    registered types, i.e. the logging application publishes its type 
information
    to the logging server.
* The logging server accumulates and merges all the type information from
   all the different sources.
* The logging server persists its own "super-symbol-table" to reduce
    traffic and also to enable the display of type information when no
    logging applications are connected.

This might be termed a "full implementation" of logging. Some high-end 
requirements
have certainly dragged it a long distance from where it started. Some 
possibly
useful lessons picked up along the way;

* The software machinery that provides for creation, transport, queueing, 
storage,
    retrieval and viewing of logs is far more important than a particular 
set of attributes.
* There is no ultimate set of attributes. The set of attributes on this 
project changed
    after almost every discussion and remains in the state "to be 
determined".
* Many of the initial requirements were eventually shown to be silly. Even 
now it remains
    very difficult to convince interested parties of this, e.g. a live 
window of
    received log records is not useful.
* There are huge benefits in creating a logging system that transports and 
stores generic
    blobs (mixed-type log records). All the significant related  software is 
"logging application"
    independent and inherently "forward compatible".
* The cost in software complexity and dependence on formalized communcations 
and storage
    mechanisms is non-trivial.
* Given another opportunity and certainly in the case of a Boost.Log 
implementation I
    would probably swing away from the "published type info" approach and go 
for straight
    text. This is mainly due to the amenable nature of text (much easier to 
get a google search
    implemented) and general software simplicity. The costs include space 
resources (RAM consumed
    in log queues and disk space) and some loss of viewing+filtering 
performance.

Cheers.