Re: [boost] Boost Logging library - first draft

From: Andrey Semashev (andysem_at_[hidden])
Date: 2007-04-02 16:58:53

Hello JD,

...Beware of the long post...

Monday, April 2, 2007, 10:11:20 PM, you wrote:

> There is no way to extend my library as basically it offers all what I
> need and look for from a logging library! :) No kidding.

You're lucky, but we should remember that there are people who would
want something more than that.

>> 1. The whole design looks more like a stream dispatcher rather than a
>> logging library which has to be more complex. I see no scope logging

> Ok so if I understand, and correct me if I'm wrong, a scope logging 
> facility would be some kind of macro added at the beginning of a 
> function. This macro holds an object (RAII style) that would log the 
> function name in its constructor (with eventually arguments) and log the
> exit of the function in its destructor. I'll add it.

Not necessarily function scope, not necessarily macro and not
necessarily actually log scope entry/exit. :) The key feature is to
maintain a loggable call stack that comes visible at each log record.
But in principle you are right - it's a scope guard that tells the
logging library the name of the current scope.

>> support, no custom log record attributes, no filter or wide logging
>> support, and I see no obvious way to implement these on the user side
>> to plug them into your lib. The way to completely wipe out logging
>> from the user code during compile time would be good too, though it's
>> not very difficult to implement it.

> - Log record attributes (what is it? just to be sure of what we are 
> talking about.)

I guess, you understood me as you have mentioned below. But to be
clear, an "attribute" is basically a piece of information of which the
logging record consists. Attributes may have different types
(integrals, strings and more complex types). Some examples of
attributes: log severity level, log origin, log record text, the
call stack described above. I believe, Gennadiy meant something like
attributes in his review when he described "key words", but the
attribute concept seems to me more generalized.

Since each entity (e.g. class, subsystem, module, thread, process...)
in a user's application may have its own set of attributes, it is
important that the logger (the "source", in general) must not be a
singleton. It must be able to represent the entity's set of
attributes. For example:

class A
{
  logger m_log;

  A() : m_log(origin("class A")) // set the log origin attribute
  {
  }

  void do_smth()
  {
    log_scope scope("A::do_smth");

    // Here we'll form a record which has at least 3 attributes:
    // - the "origin", string, equals "class A"
    // - the call stack with "A::do_smth" on top of it
    // - the message "I'm doing something"
    // There may be other attributes, such as date/time,
    // record number, thread id/name, source file name and line, etc.
    m_log.strm() << "I'm doing something";
  }
};

class B
{
  logger m_log;

  B() : m_log(origin("class B"))
  {
  }

  void do_smth()
  {
    log_scope scope("B::do_smth");

    // ... all the same, but the set of attributes is different
    // At least origin and scope attributes are different, but it
    // doesn't mean that B cannot have completely different set of
    // attributes, nor does it mean that it can't have the same
    // attributes that A has.
    m_log.strm() << "I'm doing something too";

    A a;
    a.do_smth(); // Here A logs its own attributes, not B's
  }
};

This approach gives another useful feature. We can now associate the
different entities interaction. If we had not only logger-specific
attribute set but in addition a thread-specific one, we could add some
attribute to it right before the call to A::do_smth. See:

class B
{
  void do_smth()
  {
    log_scope scope("B::do_smth");

    // ... all the same, but the set of attributes is different
    // At least origin and scope attributes are different, but it
    // doesn't mean that B cannot have completely different set of
    // attributes, nor does it mean that it can't have the same
    // attributes that A has.
    m_log.strm() << "I'm doing something too";

    thread_attribute guard(
      /* attribute  name */ "job",
      /* attribute value */ std::string("Very important job"));

    A a;
    a.do_smth(); // Here A logs its own attributes, not B's,
                 // plus a new attribute "job" we defined above.
                 // Later, when reading logs, we may see that
                 // A::do_smth was called as a part of this
                 // "Very important job" action, not something else.
  }
};

> - filter

The filter is basically a predicate which takes a record (or we may
say, a set of attributes) and tell, wether this record should be
passed to output. Filters are allowed to be connected in more complex
filters. Each filter may be targeted to one or several attributes. For
example, one filter may be checking that the record severity is higher
than 2, the second one checks that the record is generated within a
thread named "Main", the third requires that the message text matches
regexp "/Warning.*/", and the record passes to output only if the
second and either of first or third filters are satisfied.
I see a wide field of Boost.Lambda and Boost.Bind in conjunction with
Boost.Function exploit here.

And as I've said before, these filters should be able to be applied in
two phases: globally, at the library core level, and on per-sink basis.

> - wide logging (I guess it means internationalization (let's say i18n))

Generally, yes. At least the ability to log wide strings
(std::wstring) without conversion to narrow ones (std::string) should
be provided. IMHO, a reasonable compromise should be reached in this
part since a mixed-mode logging library, especially with multibyte
support, would be too inefficient in a general-purpose use case. So I
think there should be a compile-time library configuration policy to
select the native char type.

>> 2. The logger object is a singleton which, I think, would be
>> inappropriate once I would have custom attributes. I think the logger
>> should be separated from the core library, which indeed would be a
>> singleton. This would not only make attributes possible, but form a
>> "source" concept. A logger would be an implementation of this concept,
>> though allowing other implementations of "sources" (like, a
>> third-party application console interceptor, for example).

> For me the logger is the core of the logging library.
> So the source is the logger. It's the object who is responsible to 
> collect logs generated by the user? Does he format them?

Yes, the logger collects the record, though, as I exposed my view
above, it shouldn't be the core. It does not format the collected
data, it's sinks' task, since each sink may represent log records in
different way. One sink may redirect to console, the other one may
write XML log file and the third one may write a binary
zlib-compressed archive.

>> 3. The "sink" (or "target") concept is simplified down to a stream,
>> which is not a best choice. Remember that the log record is more than
>> just a string. It consists of a number of parts (attributes) - record
>> number, time, source code location, severity level, and (yet another
>> attribute) message text. There may be other user- or library-defined
>> attributes. And the "sink" should be able to define which attributes
>> to log and how to format them. In fact, a "sink" may apply its own
>> filters to wipe out the unneeded records from the output. This is not
>> possible with a simple stream "sink".

> Ok it's clearer now. An attribute is what I call the element of the log.
>   Well I'm using my element_list as a description of the format of my 
> logs. In doing such thing as:

"[" >>> boost::logging::level >> "] - " >> boost::logging::filename

I was thinking of something similar in my lib to implement text-based
sinks, though I imagined something more lambda-like (it's just a
pseudo-code yet):

sink.format() << attr("sn") << "[" << attr("date") << "] " <<
attr("message");

> I am defining the format "attributes" of my logs. Ideally, user can
> define any class deriving from log_element and implementing the toString
> function for output.

I think the logic should be opposite - the sink decides how to
represent the record, so toString is not appropriate here. I think,
some kind of visitor is needed here.

>> 4. The library is header-only, which will raise some problems in a
>> complex applications that consist of more than one module. The logger
>> instance will be visible only in one module, other modules will have
>> their own loggers, which obviously mean problems with their
>> initialization (which streams to add as "sinks"?). I believe that
>> logging library should be implemented as a shared lib.

> So, let's talk concrete. If you have two lib:
> my_library.lib
> my_other_library.lib
> main.cpp

> Including the header in each library will lead to having two differents
> instance of a logger in my final executable?
> The header-only solution seems so great to me. Include and forget...

Let's make it complicated:

my_library.dll
my_other_library.dll
main.exe

All three modules use your logging lib. How would you initialize it?

>> If you are interested, I could describe my view of architecture of
>> such logging library (I'm working on the implementation on an early
>> stage). Though, I think, many of us already have such view in our
>> minds and maybe have it implemented.
>> 

> Please, if this library is mature enough, submit it!!

As I've said, it's on the early stage. I need some time to finish the
implementation, not to mention docs and tests.

>> And finally some implementation-regarded notes (though I understand
>> this was just a draft, I couldn't hold myself, sorry :) ):
>> 1. The logger is default-constructible from the user's code, but it
>> seems that you intended to make it singleton.

> I'm calling the default constructor in the get_instance.

But get_instance is a member, as far as I can see. The constructor
should be private or protected. Otherwise you have no singleton.

>> 2. logger::get_instance is not thread-safe - function-scope statics
>> are used.
>> 3. You should not define namespace bl = boost::logging; in the library
>> code.
>> 4. Element global operators should be in boost::logging namespace.
>> They will be reachable via ADL.

> Does not compile with msvc.

Hm? Which version do you try?

>> 5. Each logging record involves std::stringstream (even not
>> ostringstream) construction, formatted output copying, and
>> destruction. I'm afraid, that would be a performance impact on the
>> application.

> Yes that's a pb.
> std::strstream logger::m_str ?

I guess...

> .... I think I took this problem too lightly. We shall not focus on
> implementation at first but more on the requirements. Austin Bingham, in
> an other post, was also saying that we should first write the minimal 
> requirements of the Boost.Logging library on some wiki or whatever... On
> the other hand, from the profile I draw in my summary earlier, 
> Konstantin mention log4cpp (http://log4cpp.sourceforge.net) as a 
> potential candidate. And still, some people propose their homegrown 
> libraries... Like everybody here, I have a limited amount of time. I was
> wondering if it would be possible to assemble a small team led by some
> boost authoritative guy (Gennadiy?) to pull this.

As I said, I'm very interested in such lib and would be glad to
participate.

-- 
Best regards,
 Andrey                            mailto:andysem_at_[hidden]