Re: [boost] Boost.Log formal review upcoming

Subject: Re: [boost] Boost.Log formal review upcoming
From: Daniel Larimer (dlarimer_at_[hidden])
Date: 2010-03-01 14:44:32

On Mar 1, 2010, at 1:03 PM, Matthias Vallentin wrote:

> On Mon, Mar 01, 2010 at 12:03:57PM -0500, Daniel Larimer wrote:
>> If you change the abstraction to nothing but a stream of data packets
>> of "some type" that flows from "some source" through various "filters"
>> to "one or more" destinations, then a "log file" becomes little more
>> than a "buffer" in the stream.  
>> 
>> Thus, if you have some "real time monitoring tools" that can be hooked
>> up to debug then these tools should be agnostic to whether or not the
>> channel they receive their data on is from the original source or from
>> the other side of a "log file filter".  
> 
> It looks like the replay tool performs some sort of input parsing on a
> given stream. What I am trying to understand is whether you are asking
> to couple those two processes. That is, do you advocate that the
> definition of the output format should automatically generate a parser
> (e.g., via Spirit) for the output? 

Lets assume you have a data source (boost::signal) and you connect it
to a chain of filters that add meta data and ultimately a filter that converts 
the types into an archive and finally one that buffers "archives to disk".  Another
filter would simply take an archive and ultimately reproduce a boost::signal.

You still need to have a "filter" that can convert a binary or text log entry 
back into the original types, but the process of fetching individual log entries and
controlling how and when they are "emited" could be entirely handled.

Now if the logging library loses all concept of "record boundary" when it writes to
disk then there is not much you can do except create a custom log reader that
determines the boundary of a specific record.  

But I contend that for many people they simply want to say "log this signal to this file under this name" and then 
open their log and say "replay signal with name to this slot" and then iterate through
the log entries possibly in a multiple of real time.  This opens up a ton of existing code which is already
separated into "producer / consumer" or "model / view" paradigms via signals to logging and visualization of logged data without
adding any new dependencies to the original code.

I do not mean to put my requirements on this if my requirements are "out of scope" or to insist that such a feature is necessary out of the box, only that a boost logging library should be flexible enough to support some means of iterating over records in a log file.

Another reason to support this kind of function is when logging a large amount of data at high data rates.... binary files would be an order of magnitude smaller, result in less disk access, and use less "formatting" overhead than text files.  Yet it should be "trivial" to convert a binary log to a text log simply by using the    source => serialization =>  binary file  => binary to obj =>  text sink. 

If your library assumes that ostream<< is defined for every type then making an assumption on boost::serialization to support such a feature would be a relatively minor addition.  

If my "requirements" fall outside the scope of 'logging' then perhaps we should define what logging means.

One last consideration is that often there is a desire to separate logging into a separate process that can
optionally "connect" to the logging core and possibly perform some additional filtering on the other side of
a socket.  

So my question is, how hard would it be to accomplish some of my requirements with the built in functions vs 
how much would I have to write from scratch to support it?  I will look into it more, but perhaps someone
more familiar with the package could offer an "estimate" in terms of hours required to implement some or
all of the features I suggest.  If they fit nicely into the framework, then I would likely contribute my own time to such an effort.  If the framework starts to "get in the way" then perhaps we should consider the design.

By the way, the library looks like it has great potential to manage debug print statements when layered with appropriate macros.  

> 
> In general, I would love to see the library in Boost. My logging
> facilities are built on top of the current SVN version of the library
> and became an invaluable and flexible tool throughout my development.
> 
>   Matthias
> -- 
> Matthias Vallentin
> vallentin_at_[hidden]
> http://www.icir.org/matthias
> _______________________________________________
> Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost