[Log] Formatting multiple attributes in a single formatter

From: Michael Jones (mike_at_[hidden])
Date: 2020-08-24 19:18:10

TL;DR; I'm looking for help/advice/code on creating a custom
formatter, or formatter factory, to format more than one attribute at
a time into a single "placeholder" in the format expression. I do not
yet have code, working or otherwise, to try to solve this, because I
am still trying to understand the design space of Boost.Log's API.

I have been using Boost.Log successfully in my project for about 1-2
years now, with many hundreds of log statements of the style

MY_LOG_MACRO(Normal) << "My Log Message"
    << std::endl << "- Attribute 1:" << myAttributeVar1
    << std::endl << "- Attribute 2:" << myAttributeVar2;

I now want to convert these hand-formatted attributes that are
directly inserted into the message into attributes in the Boost.Log
meaning.

However, these attributes are "free form", in the sense that they are
not pre-registered with Boost.Log before use (which is obvious given
they are manually formatted into the message string), and they do not
conform to any specific naming conventions or value-type scheme. I
estimate that there are a few hundred different attributes that are
printed to my logs in various places as "one off" attributes that are
not really meaningful outside of the specific log statement they are
used in.

My current plan is to provide my developers with a way to
incrementally move their logging toward using Boost.Log's attributes
without requiring a flag-day where every log statement is changed in a
single merge. I want to do this by allowing these attributes to be
added to the log message using boost::log::add_value() from
boost/log/utility/manipulators/add_value.hpp, such as:

MY_LOG_MACRO(Normal) << "My Log Message"
    << boost::log::add_value("Attribute 1", myAttributeVar1)
    << boost::log::add_value("Attribute 2", myAttributeVar2);

However, I want to provide the same formatting semantics that they are
used to, to make the transition easier.

At this stage, it is not clear whether I'll ever be able to get a set
of standardized attributes that can be pre-registered with the
Boost.Log library. A surface level estimate shows me that I have, at
the minimum, many dozens of different attributes with unique names /
value-types. This would make addressing each such attribute name
individually prohibitively expensive to add to the format expression

Currently, using boost::log::import_from_stream(), I use a format
expression like:

    Format="%Process%/%Channel% - [%Severity%] %Message%"

And I would like to change that to the following, or something similar:

    Format="%Process%/%Channel% - [%Severity%] %Message%\n%Attributes%"

The intention of %Attributes% here would be a placeholder in the
format expression that gets resolved to a custom formatter, or
formatter factory, which will iterate through the list of attributes
in the current record_view and print each attribute name and value
with a new line separator.

Ideally, this formatter or format factory would be able to recognize
that a particular attribute has already been printed, or will be
printed, so that it only adds new information to the log. However, I
believe this is not something that can be easily done, so I currently
plan to have this duplicated information in the log with the intention
to resolve it another time.

One way I could resolve the duplicate information problem is by adding
an option block to the %Attribute% placeholder, e.g.
%Attribute[Exclude:Process, Exclude:Channel, Exclude:Severity,
Exclude:Message]%.  It would be ugly, but it would do the job.

So far I've gathered the following information about this subject:

How the default formatter factory works by creating a
default_formatter, which has special handling for various time types,
but otherwise just calls operator<<(stream_type, value);
https://github.com/boostorg/log/blob/develop/src/setup/default_formatter_factory.cpp

How the format parser works by, well, parsing the format expression to
get attribute names and other important information for later use. I
don't have a complete understanding of this, but I get the gist of it.
https://github.com/boostorg/log/blob/develop/src/format_parser.cpp

How sinks_repository from init_from_settings.cpp works by pre-defining
multiple different sink-factories with the repository, and then
init_from_settings() uses those pre-defined factories to support
loading the settings. Notably, I use default_text_file_sink_factory,
and default_console_sink_factory in my own code. Both of those sink
factories just use boost::log::parse_formatter() to create the
formatter from the format string in the settings file.
https://github.com/boostorg/log/blob/develop/src/setup/init_from_settings.cpp

How formatter_repository and works by storing a map between the
Attribute name and the Formatter Factory to use.
https://github.com/boostorg/log/blob/develop/src/setup/formatter_parser.cpp

Record_view, it's read-only nature, and how to access the attributes
for the record using the boost::log::record_view::attribute_values()
function.
https://github.com/boostorg/log/blob/develop/include/boost/log/core/record_view.hpp

The next thing I am going to try is to define a custom formatter for
the %Attributes% attribute name, and pre-define some constant value
for it so that it is always defined for each record. Then I'll have a
formatter factory like this (psuedo code, not compile tested)

template< typename char_type >
struct abitrary_attribute_formatter
{
    void operator()
    (
        boost::log::record_view const& rec,
        boost::log::basic_formatting_ostream< char_type >& strm
    ) const
    {
        for(auto const& attr : rec.attribute_values())
        {
            strm << "- " << attr.name() << ": " << attr.value();
        }
    }
};

What I'm currently trying to understand is how I can create a custom
formatter that dispatches to other formatters. How is this done?

Is there a more straight-forward way to accomplish the goal of
providing a formatter for the concept of "All available attributes"?
If not, can anyone offer implementation or design advice?