- Boost - lists.boost.org

Re: [boost] [log] Boost.Log Review...
by Sean Chittenden 14 Mar '10

14 Mar '10

>> The above wasn't explicitly distilled in the documentation in a >> single place, but I think future developers would find it very useful >> to see that spelled out explicitly. Given the extremely modular >> design of the library, I understand why the docs are the way they are >> and know that some of this is touched on in the Design Overview, but >> until I became very competent with the library, the documentation >> seemed too abstract. Despite the importance of logging, becoming a >> logging expert was tangential to the problem I was trying to solve >> and I would have personally found the above summary very useful. > > Well, that summary may describe your use case very well, but it is by no > means complete with regard to Boost.Log as a whole. I would even say > it's misleading in some parts - in particular: "Loggers are collections > of Sinks" and "Think of channels as a Sink-level observer". Fair response. I think a more general way for me to classify my response/concern (instead of providing a loosely worded example) would be: "Which class does what and how do the classes relate to each other?" Right now answering that question results in a decent exploration of Boost.Log's code and documentation. It would have been useful to me, and likely future developers, to have the classes that comprise Boost.Log summarized so that conceptual map could have been more easily formed without having to dig in to the actual headers. Something similar to the contents of the Definitions page, but with architectural clues. http://boost-log.sourceforge.net/libs/log/doc/html/log/defs.html As an example, expanding on the Sinks definition from above: > Log Sink: A target, to which all log records are fed after being collected from user's application. It is sink's nature that defines where and how the log records are going to be stored. Maybe include something as brief as: > With a Log Sink, developers can modify the filtering and add additional backends. I consider myself a modestly competent user of boost, but still am getting parts of this wrong after over three months of using/digging in to Boost.Log. >> A hindsight 20/20 architecture and terminology for Boost.Log could be >> simplified in terms of: > > What is "20/20"? I'll continue my answers assuming that the following > was your understanding of the current Boost.Log architecture. Correct. The contents of this email has evolved from sometime around mid-January or whenever I learned that Boost.Log was going to be reviewed and accepted. It also crudely mirrors my understanding of the library as I went through the development process of utilizing the library. >> *) Log Record pre-conditions. Sinks control whether or not a Log >> Record is to be generated > > Looks like you're confusing sinks with filters. Also, don't forget about > the global filter. I became sloppy with my terminology. Filters are stored in Sinks and the Global core. Speaking of filters, yesterday I stumbled across a feature request: > Program received signal SIGABRT, Aborted. > 0x00007fff81de2fe6 in __kill () > (gdb) bt > #0 0x00007fff81de2fe6 in __kill () > #1 0x00007fff81e83e32 in abort () > #2 0x00007fff81e03de2 in _Unwind_Resume () > #3 0x00000001000a3b1c in main (argc=1, argv=0x7fff5fbff338) at /tmp/foo.cc:267 If I wrap everything in a boost exception, I get: > exception: /usr/local/include/boost/log/filters/attr.hpp(134): Throw in function (unknown) > Dynamic exception type: N5boost16exception_detail10clone_implINS0_19error_info_injectorINS_12log_mt_posix13missing_valueEEEEE > std::exception::what: Requested attribute value not found It'd be nice if the error message were able to somehow include the string representation of the value (attribute) it wasn't able to find. Turns out it was a type-o so I had an unregistered attribute, but the type-o would've been very easy to identify if it were included in the error message someplace (instead it took a fair amount of manual investigation and backtracking to figure out it was a filter searching for an attribute that didn't exist). [snip useful responses/corrections ] >> Observers could then be placed on Sinks that would stamp Log Records >> with Severities, or Channels, or in whatever hodge-podge >> configuration the developer wanted and such examples would be very >> cookbook. > > Now that you presented a code snippet, your suggestion makes more sense > to me. Let me express my thoughts on it in two parts: > > 1. Custom log records. Yes, the model of "mandatory" attributes look > promising. It could, at least, partially solve the problem of possible > library misconfiguration that was highlighted in another review. I > imagine that some kind of log record prototype (or factory) should be > defined by the user in order to be able to construct log records with > the specific set of attribute values. Filters and formatters could then > request these mandatory attribute values without having to worry about > their absence. Lookup performance may also be higher, but I doubt it > will be compile-time. The idea is quite vague now but after some > thinking over it might become a good addition to the library. > > 2. Designing sinks (and the core, I presume) after the log records. No, > that looks unacceptable to me, at least for what Boost.Log tries to > achieve. This would introduce too much coupling between the logging side > of the library (which tends to be exposed to many parts of the > application) and the consuming side (which tends to be configured in a > single place of the application). To my mind, these sides of the library > should be maximally decoupled. Also, it's not clear how to support > multiple log record types in such a design. >> *) Global loggers vs. per-object references to singletons > > Not sure I understand. Is it better to have global objects that reference singletons: > typedef boost::log::sources::severity_channel_logger_mt<levels::severity_level> logger_t; > > BOOST_LOG_DECLARE_GLOBAL_LOGGER(default_log, logger_t); > BOOST_LOG_DECLARE_GLOBAL_LOGGER_CTOR_ARGS(stats_log, logger_t, (boost::log::keywords::channel = "stats")); or per object references to logging singletons: > typedef boost::shared_ptr< boost::log::sinks::synchronous_sink< boost::log::sinks::text_ostream_backend > > console_sink_t; > typedef boost::shared_ptr< boost::log::core > logger_core_t; > > struct my_obj { > my_obj(logger_core_t& logger) : lg(logger) {} > void foo() { BOOST_LOG(lg) << "log msg"; } > logger_core_t& lg; > } > > int main() { > console_sink_t cerr_sink; > cerr_sink = boost::log::init_log_to_console( > std::cerr, > boost::log::keywords::filter = boost::log::filters::attr< int >("Severity") >= warning, > boost::log::keywords::format = "%TimeStamp%: %_%", > boost::log::keywords::auto_flush = true); > my_obj o(boost::log::core::get()); > o.foo(); > } ... I think that example's right. >> *) Channel severity loggers example > > That example wouldn't be much different from the ones in sections about > severity_logger or channel_logger. But I'll add a code sample. Where I spent time with this was figuring out how multiple chan severity loggers interacted with the core and how my app was supposed to interact with the distinct loggers, which were linked in to the core, behind the scenes. > typedef boost::log::sources::severity_channel_logger_mt<levels::severity_level> logger_t; > BOOST_LOG_DECLARE_GLOBAL_LOGGER(default_log, logger_t); > BOOST_LOG_DECLARE_GLOBAL_LOGGER_CTOR_ARGS(lg1, logger_t, (boost::log::keywords::channel = "chan1")); > BOOST_LOG_DECLARE_GLOBAL_LOGGER_CTOR_ARGS(lg2, logger_t, (boost::log::keywords::channel = "chan2")); > #define LG1_WARN BOOST_LOG_SEV(lg1::get(), log_severities::warn) > #define LG1_INFO BOOST_LOG_SEV(lg1::get(), log_severities::info) > ... > LG1_INFO << "info msg"; >> *) Setting up an initial console logger as >> line one in main() > > Hmm. Ok, I'll add something about that to the tutorial. Easy to do (see above), but I didn't see anything that said, "it would be wise to do this first and then remove your console sink after you setup your logging environment." Initially I disregarded the init_to_console() convenience functions while reading the docs, and it wasn't until I started dealing with error messages that weren't being logged to a file because my logging setup routines hadn't been completed (or had failed for some reason due to FS perms), that I suddenly took note of needing to get console logging working first. >> *) Changing a backend filter at a later date > > Just call set_filter on the backend, that's all. This was one of the areas where there were lots of ways that I could do it, but I couldn't tell which was the right way. libs/log didn't suggest a preferred mechanism or right way to do it. >> *) Using async logging for INFO level logs, but switching to sync for >> CRIT errors. > > You can register two sinks - one synchronous and one asynchronous, and > set their filters accordingly. Ah, clever way of handling that. >> *) Obtaining the severity of a particular logger > > You mean, the default severity? There's no public API for this, but > there's a protected function default_severity in the logger feature. Why > would you need it? Internally you're using spirit for log record filtering, so it's impossible from what I can tell to do that inside of Boost.Log. Having a way of poking at a sink to obtain information about its runtime status. Using the above console sink as an example: > typedef boost::shared_ptr< boost::log::sinks::synchronous_sink< boost::log::sinks::text_ostream_backend > > console_sink_t; > console_sink_t cerr_sink; > cerr_sink = boost::log::init_log_to_console( > std::cerr, > boost::log::keywords::filter = boost::log::filters::attr< int >("Severity") >= warning, > boost::log::keywords::format = "%TimeStamp%: %_%", > boost::log::keywords::auto_flush = true); > > std::cout << "Current Severity Level: " << /* or something like this */ cerr_sink->filter_severity() << std::endl; I ended up adding this to my application's surrounding code. >> *) How do you remove log events given a severity level at compile >> time. Currently this isn't explicit defined in the documentation >> like it is in Google's glog library: >> >> http://google-glog.googlecode.com/svn/trunk/doc/glog.html#strip > > The library has no direct means for that. Users can conditionally define > their logging macros, based on BOOST_LOG_SEV or pure C++ interface and > then enable or disable the particular logging statements from the source. *nods* I just didn't see anything in the docs to do this, which is why I ended up with the practice: > #ifdef RELEASE > # define LG1_WARN BOOST_LOG(null_log::get()) > #else > # define LG1_WARN BOOST_LOG_SEV(lg1::get(), log_severities::warn) > #endif null_log always filters everything sent to it. Probably less than ideal, but I couldn't think of a better way to handle this. >> *) Is there a way to set a bit in a Log Record that overrides a >> Sink's filter? For instance, there are certain events that I want to >> make sure go to the console 100% of the time without changing the >> severity of the log message. Do I just log twice? It'd be keen if >> Log Records could have a bit that could be set and would override a >> filter. >> >> bool log_to_console = true; BOOST_LOG_SEV_CONS(lg, warning, >> log_to_console)<< "my message"; > > You can set an attribute (say, "Emergency") for that record, and setup > filters so that whenever this attribute is present, the record is passed > on to the console backend. You mean a scope attribute? > boost::shared_ptr< attrs::mutable_constant< bool > > console_attr(new attrs::mutable_constant< bool >(false)); > lg.add_attribute("Console", console_attr); > BOOST_LOG(lg) << "Console is false by default"; > > { > BOOST_LOG_SCOPED_THREAD_ATTR(lg, "Console", true); > BOOST_LOG(lg) << "Console is true"; > } > > BOOST_LOG(lg) << "Console is false again"; Like that? >> *) It should be noted that the internal async log message queuing >> mechanism uses a vector and during very bursty logging activity that >> the vector can grow large in size and doesn't ever reduce in size. >> This is a known limitation and the use of a vector was chosen to >> facilitate atomic append operations for log records. Using a >> std::deque would be preferred for bursty applications that way space >> could be reclaimed easier, but as Andrey pointed out, it would >> require a full mutex which would reduce the performance of the >> library. > > There is no vector in the async frontends. The problem with the current > queuing mechanism is not the storage strategy but the lack of queue > boundaries. But I agree, I should have drawn attention to this aspect in > the docs. Will do. *nods* I brought this up in a private email w/ you and was satisfied with your answer, but thought it worth noting for others. As for the frontends having their own queue, with the way that async frontends work and their feed_records() call, I'd assumed the backend used a time-based mechanism to come by and pick up the log records stored in per-frontend TSS queues. Something where frontends atomically pushed records on, and the backend would atomically swap() the list of records off for sorting and processing by the backend. Just a point of confusion after reading: http://boost-log.sourceforge.net/libs/log/doc/html/log/detailed/sink_fronte… and I never dove in to the source to investigate further. >> *) I find the argument against Lazy Streaming of attributes to be >> weak. >> >> Current BOOST_LOG() expands to a for() loop where the contents of >> the for() loop aren't executed unless a Sink in a Logger accepts the >> record. >> >> So why can't Log Records have compile-time attributes that can be >> populated with typed attributes in a stream? >> >> And then be able to extract the compile-time attribute in a backend: > > Just as it is said in the rationale - the streaming expression is _not_ > executed until _after_ the filtering is done. And filtering already > involves attribute values, so there's no way to use these attribute > values you specified in the streaming expression. Besides, after the > view of attribute values is constructed, it is immutable, so nothing can > be added after the filtering is done. That's why I make the distinction between "log record preconditions," attributes that can be used in a filter, and "attributes," which is a catch-al term for data stored in a Log Record. I agree that it is impossible for streamed attributes to be used as preconditions, but streamed attributes act as an extremely convenient (from a syntax perspective) way of passing data to a backend for processing. >> *) Dovetailing on the above two points, a flush bit for a given Log >> Record would be very nice as well. 99.999% of the time async logging >> is desired, but in the 0.001% of the time when a Log Record must get >> out to disk, chances are that one Log Record every blue moon is >> disproportionally valuable compared to the normal Log Record. Having >> a flush() that doesn't return until the backend has fsync(2)'ed data >> would be really spiffy. >> >> BOOST_LOG_SEV(lg, warning)<< "My message"<< lg::flush; > > As I already noted, your mileage is too biased, at least. I, personally, > tend to use synchronous logging. But I do recognize that some kind of > flushing ability would be nice in case of async logging. I like the idea of using both async and sync sinks. It leaves me with a few questions, however: How does that work if they're both using the same text file backend? Will log entries still be ordered? > typedef boost::log::sinks::text_file_backend backend_t; > typedef boost::log::sinks::ordering_asynchronous_sink< backend_t > sink_t; > boost::shared_ptr<sink_t> sink(new sink_t( > boost::make_shared< backend_t >(), > keywords::order = logging::make_attr_ordering("LineId", std::less< unsigned int >()), > keywords::ordering_window = boost::posix_time::seconds(1))); Thanks again in advance. -sc -- Sean Chittenden sean(a)chittenden.org

2 1

[SoC] Summer of Code Project Ideas
by Andrew Sutton 13 Mar '10

13 Mar '10

I'm officially soliciting ideas for student projects for this year's 2010 Summer of Code. If you're maintaining or contributing to a library and have some non-trivial items on a TODO list, then suggesting that as a possible student project may be a viable idea. Unlike the previous years, I think that proposing work on sandbox libraries would also be a good idea. There is a substantial amount of work in the sandbox that might be well-suited for student projects and eventual (I hope) acceptance to Boost. In general student projects should be tightly focused with a clear set of goals to getting their work integrated into a mainline development branch (be that trunk or sandbox). In general, we've noticed that ground-up libraries almost always fail to be integrated or released independently since they require a commitment longer than the summer. There are, of course, notable exceptions to this rule. If you are going to propose a student project, please keep in mind the following things: 1. Students tend to go away in the fall. 2. If their work isn't tended after August, then it's largely forgotten 3. If the student is gone and their work is forgotten, then the project didn't really succeed. If you write down or submit a project, please include your expectations for that work. And try to keep it small. That said, if you're too precise in the specifications, we'll going to get a couple dozen proposals that simply reiterate your expectations. But that's the nature of the beast. Good ideas for projects might be: == Data Structures and Algorithms== Admittedly these have little impact on overall quality of Boost, but make a nice sandbox for students to work in and can result in surprisingly good code. Over the course of the summer 1 or 2 related data structures or a family of related generic algorithms seems to make a nicely packaged project. ==Extensions for Existing Libraries== There are a number of Boost libraries that are very amenable to extension (e.g., BGL, GIL, Math, etc.). Basically any library that doesn't have a finite feature space. Non-intrusive changes (i.e., those that don't require hacking on the existing bits) typically make good summer of code projects. == Infrastructure Projects== Projects that aren't part of Boost, but help us improve the quality of Boost might make decent projects, but we've never had one proposed or accepted that I'm aware of. Remember, we can't accept documentation projects so "Write my documents" will not be a good student project (unfortunately ;) ==Applications of Boost== This is a little different... It might be a good idea to have students build real (example?) applications that use the Boost C++ Libraries. There are a number of real benefits to having students work in this space. First, these applications are very obvious clients of Boost. They can provide immediate feedback on issues with the interface, usability, correctness, performance, documentation, etc. Second, they might make really good examples of best practice for using Boost. Third, they probably make nice student projects. If you have any ideas, let's hear them. And remember, funded projects need mentors. Andrew Sutton andrew.n.sutton(a)gmail.com

10 42

[log] Boost.Log Review...
by Sean Chittenden 13 Mar '10

13 Mar '10

YES, the Boost.Log library SHOULD BE ACCEPTED in to the Boost release distribution. Summary: *) Boost.Log isn't perfect and after having reviewed most of its alternatives, and having recently converted from using glog, I like the utility and architecture provided by Boost.Log the most. It comes close to solving all logging problems for everyone. Closing the gap of solving the last 5% between what Boost.Log does and what I ideally wanted was tedious and required me to develop a much higher level of understanding that I would have liked, but it was possible. *) In solving all logging problems for all people, Boost.Log's documentation is quite abstract. *) I would like to see the design of templates/classes flipped so that the starting point for interacting and beginning to use Boost.Log begins with customizing Log Records and ends with defining backends and backend filtering. This is a fundamental design choice, however and one that I'm willing to accept given the value that Boost.Log provides. *) I've been saving most of this feedback for the better part of two months now so excuse the length and rambling. Certain points of the library I struggled with and ultimately ended up begrudgingly resigned myself to accepting as "the way it is" vs. "the way it should be." *) My perspective on the Boost.Log is framed by the recent use of Google's glog library and in a previous life having written performance-oriented, binary logging libraries for high TPS systems (only in C, however, not C++). > - What is your evaluation of the potential usefulness of the library? Boost.Log quite useful and has the highest promise of any logging library that I am aware of because of its ability to accommodate the following common use case: Support asynchronous logging from multiple frontend threads with a backend able to order the output and send different messages to different files. If it weren't for the ability of this library to meet that specific design requirement, I would still be using Google's glog library. > - What is your evaluation of the design? The design is very modular. Too modular. So much so that the flexibility and required understanding is actually a barrier to entry to make use of the library (i.e. now that I'm using the library, I know more about the innards of Boost.Log than I would like or think is reasonable). After using this library for several months now, I have distilled a quick summary for those coming up to speed on how to use Boost.Log and its components. This is specific to how I'm using Boost.Log, but I think this is the desired use-case for most users of Boost.Log (esp since this is the common path for users of ASIO). - Your application is littered with Log Statements that are tagged with different Severity Levels - Log Statements consist of a Logger (a collection of Sinks), a Precondition (e.g. a severity) and Log Record instance data (most commonly a message) - Loggers are collections of Sinks (e.g. a file and the console) - Loggers conditionally accept Log Records if one of a Logger's Sinks returns true for a Precondition (e.g. only create a Log Record if the log statement's Severity Level greater than or equal to the Severity Level configured in the Sink) - Instructions required to create a Log Record won't be executed if a Sink doesn't accept a Log Record - Sinks can be configured to queue accepted Log Records (e.g. async frontends) before passing them to one or more Backends for printing (e.g. console text file, SNMP trap) The Boost.Log system can be configured to mutate individual Log Records through the manipulation of the following three types of information: - Channels represent a way of tying together a collection of related systems (e.g. security, network, application). Think of channels as a Sink-level observer that stamps the channel on a Log Record. - Severities represent the importance of a message (a Log Statement precondition that a Sink-level Observer could stamp a Log Record) - Attributes represent per-Log Record information (a string key/value in a Log Record instance) Backends can then format the output based on the Attributes of a particular Log Record. The above wasn't explicitly distilled in the documentation in a single place, but I think future developers would find it very useful to see that spelled out explicitly. Given the extremely modular design of the library, I understand why the docs are the way they are and know that some of this is touched on in the Design Overview, but until I became very competent with the library, the documentation seemed too abstract. Despite the importance of logging, becoming a logging expert was tangential to the problem I was trying to solve and I would have personally found the above summary very useful. While glog is a supper thin logging API, Boost.Log feels like a complete and modular logging system. In glog, seeing: VLOG(3) << "asdf"; Where 3 can/was a #define, the system ended up feeling very arbitrary and unstructured. A hindsight 20/20 architecture and terminology for Boost.Log could be simplified in terms of: *) Log Record pre-conditions. Sinks control whether or not a Log Record is to be generated *) Loggers. A container for zero or more Sinks *) Sinks. A frontend that queues Log Records *) Sink Compile-Time Attributes/Observers. Observers that are configured at compile time, augment Log Records, and stamp Log Records as they pass through the Sink (e.g. severity, channels, etc) *) Backends. A formatter that orders and writes Log Records from Sinks *) Log Records. A container of information that is populated by the user's application, Sinks and Backends (possibly Loggers as well) *) Log Record Compile-Time Attributes. Compile-time members of a Log Record struct *) Log Record Runtime Attributes. Key/Value members of a Log Record stored in a std::map (currently how Attributes are implemented) I'd imagine most developer would want async messaging between Sinks and Backends. Those that don't, could make use of a Sink Compile-Time Attribute that always sets the flush bit on a Log Record. > - What is your evaluation of the implementation? Generally speaking the internals of the implementation is pretty clean, especially given the size and various ways that Boost.Log can be cobbled together. Also in hindsight: the developer interface for interacting with the library could be improved by making Log Records the fundamental unit that everything hangs off of. Making log records easy to extend containers that carry domain specific information without std::map seems like a better design choice given how frequently developers will interact with Log Records. The use case of having log records that maintain stats that get pumped through a logging system struck me as particularly useful, but unnecessarily expensive with std::map. As was noted in other reviews, the cost of evaluating whether or not a Sink is going to accept a Log Record is of some concern. Figuring out a way to have the Log Record preconditions evaluated as quickly as possible should be important given the frequency at which logging code is/should be deployed in most applications. I remember reading someone else's benchmark saying that it was ~2500 instructions to evaluate whether or not a Sink should accept a given Log Record. For something that should macro expand to: for (logger::iter = logger.begin(); iter->accept(severity); ++iter) iter->stream() I haven't profiled, but when most accept() functions are doing: inline bool accept(SevT s) const { return s >= sev_filter ? true : false; } 2500 ops seems expensive to me. The notion of a "global logging core" bothers me. Having singletons of loggers and sinks in TSS is very good from a utility perspective. Admittedly I don't know how the global logging core is used beyond being a nob that can be turned to enable/disable logging, nor have I profiled it yet, but any centralized core bugs me in a highly threaded environment. Questions: Why does Boost.Log roll its own TLS/TSS support? boost/log/detail/thread_specific.hpp vs. boost/thread/tss.hpp There may be a very good reason, I just didn't see it when I was looking through the library. Can Log Records be pool managed somehow? *) Log Records I understand why std::map is used to store attributes (ease of use/flexibility), but at the same time, I do wish that frequently stored attributes could be baked in at compile time. This is my biggest gripe with the library, actually. I would like my application to err on the side of making use of compile-time information at the expense of the time required for compiling the source. Storing things in a std::map seems lazy and brutish, for the lack of a better description. Flipping things on its head so that Log Records define Loggers, Sinks and Backends would be ideal, IMHO, and would solve most of my gripes. This is a design decision, but the contents of Log Records is of more importance than the backends. Boost.Log appears to be designed from Backends to Log Records vs. Log Records to Backends, which is the way data flows through an application (this would be my preference). For example: // quick pseudo-code using MI to create my_log_record and downstream types typedef enum { info = 0, warn = 1, crit = 2 } severity_t; struct my_log_record : boost::log::record::attribute::map< std::string, std::string >, boost::log::record::attribute::severity< severity_t >, boost::log::record::attribute::message, boost::log::recordmy_app::log_record::attribute::src_port {}; typedef struct my_log_record log_record_t; typedef SinkT< log_record_t > sink_t; typedef LoggerT< sink_t > logger_t; typedef BackendT< sinkT > backend_t; Observers could then be placed on Sinks that would stamp Log Records with Severities, or Channels, or in whatever hodge-podge configuration the developer wanted and such examples would be very cookbook. *) Log Records are highly reused and seem like ideal candidates for a TLS freelist. > - What is your evaluation of the documentation? Pretty complete, albeit abstract. Lacks examples for customized records and a best practice guidelines. A few other suggestions for things that I had to hunt down or figure out: *) Global loggers vs. per-object references to singletons *) Channel severity loggers example *) Setting up an initial console logger as line one in main() *) Changing a backend filter at a later date *) Using async logging for INFO level logs, but switching to sync for CRIT errors. *) Obtaining the severity of a particular logger Outstanding doc questions: *) How do you remove log events given a severity level at compile time. Currently this isn't explicit defined in the documentation like it is in Google's glog library: http://google-glog.googlecode.com/svn/trunk/doc/glog.html#strip *) Is there a way to set a bit in a Log Record that overrides a Sink's filter? For instance, there are certain events that I want to make sure go to the console 100% of the time without changing the severity of the log message. Do I just log twice? It'd be keen if Log Records could have a bit that could be set and would override a filter. bool log_to_console = true; BOOST_LOG_SEV_CONS(lg, warning, log_to_console) << "my message"; Other comments: *) It should be noted that the internal async log message queuing mechanism uses a vector and during very bursty logging activity that the vector can grow large in size and doesn't ever reduce in size. This is a known limitation and the use of a vector was chosen to facilitate atomic append operations for log records. Using a std::deque would be preferred for bursty applications that way space could be reclaimed easier, but as Andrey pointed out, it would require a full mutex which would reduce the performance of the library. *) I find the argument against Lazy Streaming of attributes to be weak. http://boost-log.sourceforge.net/libs/log/doc/html/log/rationale/why_not_la… Current BOOST_LOG() expands to a for() loop where the contents of the for() loop aren't executed unless a Sink in a Logger accepts the record. From: boost/log/sources/record_ostream.hpp #define BOOST_LOG_STREAM_WITH_PARAMS_INTERNAL(logger, rec_var, params_seq)\ for (BOOST_LOG_AUTO(rec_var, (logger).open_record((BOOST_PP_SEQ_ENUM(params_seq)))); !!rec_var; rec_var.reset())\ ::boost::log::aux::make_pump_stream((logger), rec_var) So why can't Log Records have compile-time attributes that can be populated with typed attributes in a stream? BOOST_LOG_SEV(lg, warning) << "My log has an attribute src_port" << log_record_t::set_src_port(conn.src_port) << " this continues to append to the Log Record's message"; And then be able to extract the compile-time attribute in a backend: // Dashed out pseudo-code to make a point (I know the syntax is garbage in the src_port's formatter). typedef struct backend : boost::log::record::base_record { typedef int src_port_type; int src_port; } record_t; typedef boost::log::sinks::text_file_backend< record_t > backend_t; b->set_formatter( fmt::stream << fmt::attr<unsigned int>("LineId", keywords::format = "%08x") << " Src Port: " << fmt::attr< record_t::src_port_type >(record_t::src_port) ); *) Dovetailing on the above two points, a flush bit for a given Log Record would be very nice as well. 99.999% of the time async logging is desired, but in the 0.001% of the time when a Log Record must get out to disk, chances are that one Log Record every blue moon is disproportionally valuable compared to the normal Log Record. Having a flush() that doesn't return until the backend has fsync(2)'ed data would be really spiffy. BOOST_LOG_SEV(lg, warning) << "My message" << lg::flush; As is, you have to call: sink->feed_records(::boost::posix_time::seconds(0)); which isn't as elegant and I don't believe actually signals to the backend that data needs to be flushed to disk (it simply drains the frontend). Developers that want synchronous logging could define an observer that always sets the sync bit and would almost certainly be optimized away at compile-time. > - Did you try to use the library? With what compiler? Did you have any problems? FreeBSD 7+ and Darwin. GCC 4.2.1, gcc 4.3 gcc 4.4. TIP in subversion has been working well for me. RC4 was problematic due to bjam configuration problems. > - How much effort did you put into your evaluation? A glance? A quick reading? In-depth study? Developed a reasonably in-depth understanding of the internals and its use. > - Are you knowledgeable about the problem domain? Sufficiently adept in the problem domain. Parting thoughts: Like I said in the first paragraph, this library should be accepted in to Boost because it's the best that I've evaluated and see available on the landscape. My compliments to Andrey, and likely Trojo as well. All of my critical feedback is hindsight 20/20 after having successfully integrated Boost.Log in to my application. I am happy with Boost.Log though do see room for improvement in the future. Forward thinking: The more times I have to tweak and revisit logging, for whatever reason, the more convinced that a better and more elegant design would result if Backends, Sinks, et al were defined in terms of Log Records. If there is a Boost.Log2, I would love to see the architecture flipped on its head as suggested. I believe such an architecture would: *) improve the evaluation of a Sink's preconditions for accepting/creating a new Log Record *) allow Log Records to contain bits that are set by observers and are propagated through a Log Record's life-cycle back to the Backends *) result in easier application specific logic (e.g. flush or console redirection) *) improve performance when preconditions are evaluated *) improve performance because fixed sized Log Records could be memory managed on thread specific pools *) allow observers to be chained together components that would be easier to setup and manipulate; and *) could be used to allow data to stream to a backend (namely application stats) Further, by constraining the scope of what Boost.Log is capable to a more rigid/fixed architecture that gets optimized at compile time, the library would be easier to understand and use (e.g. change the design so that not all Sinks or Backends accept all Log Records). Hopefully someone finds this retrospective useful in the future. -sc PS I started to put my money where my mouth is and cobbled together something that crudely demonstrates Log Records -> Backends, but came up short when getting streaming attributes to work due to a lack of time available to focus for getting an overloaded operator<<() to work. The sample can be found here if anyone is interested. Discussions of this should be taken offline since it's tangential to the current Boost.Log review process. http://codepad.org/8ACp0obH -- Sean Chittenden sean(a)chittenden.org

2 1

Status of (boost) lockfree?
by Emil Dotchevski 13 Mar '10

13 Mar '10

I decided to give the lockfree library that's currently in the review queue a try. On MSVC 9.0, I'm getting this error: boost\lockfree\detail\cas.hpp(129) : error C2440: 'reinterpret_cast' : cannot convert from 'const boost::uint64_t' to 'LONGLONG' The compiler then continues to say: Conversion is a valid standard conversion, which can be performed implicitly or by use of static_cast, C-style cast or function-style cast Am I missing a configuration macro, or is this a valid compile error? Emil Dotchevski Reverge Studios, Inc. http://www.revergestudios.com/reblog/index.php?n=ReCode

4 5

[serialization] portable non-floating-point binary library.
by Mostafa 13 Mar '10

13 Mar '10

Hi all, If adding floating point support to portable binary serialization is so hard/time-consuming, why not just have a portable non-floating-point binary library in boost as an intermediate step? Surely it would be more convenient for the community, instead of having to build in or make a library out of the files in https://svn.boost.org/trac/boost/browser/trunk/libs/serialization/example. (I did search the archives and couldn't find any answers, if I missed some threads please point them out.) Thanks, Mostafa

1 0

suggestion on assertion macros
by DE 12 Mar '10

12 Mar '10

there were a discussion recentrly about verifying arguments of assertion macros while checks are turned off i've got an idea of how to check the argument(s) and guarantee zero code size and runtime overhead suppose we have a canonical macro 'assert()' while compiling with NDEBUG macro defined it expands to nothing which may cause annoying compiler warnings (like 'var isn't referenced') assert() checks a condition i.e. the argument must be convertible to bool then we can write #define assert(cond) sizeof(cond) but to check if the arg converts to bool we can write something like #define assert(cond) sizeof(bool(cond)) which may or may not expand to some code (i.e. machine instructions) to ensure zero overhead i suggest the following scheme: #ifdef NDEBUG typename<size_t s> struct dummy_for_assert; #define my_assert(cond)\ typedef dummy_for_assert<sizeof(bool(cond))> typedef_dummy_for_assert #elif #define my_assert(cond) assert(cond) #endif i hope you got the idea by the way the typedef may be enclosed in a scope in order not to mess around: #define my_assert(cond)\ do {\ typedef dummy_for_assert<sizeof(bool(cond))> t;\ } while(false) although all this looks like a big kludge i believe more advanced programmers would make it work very well (and even make it idiomatic) if that already has been discussed sorry for wasting your time -- Pavel

6 13

[repost][multiarray] Bug with boost::detail::multi_array::array_iterator on MSVC 2010RC?
by Jesse Perla 12 Mar '10

12 Mar '10

Sorry for the repost, but I am pretty sure this is a bug with the MSVC standard library integration to multi_array. It could also be related to boost::iterator The following code worked on MSVC 2008, Intel 11.1 on Windows, and MinGW, ... but is failing to compile on the MSVC 2010RC. I tested this against boost 1.42 and the boost trunk from SVN. #pragma warning(disable:4996) //Tough to see errors otherwise. #include <boost/multi_array.hpp> int main(int argc, char* argv[]) { boost::multi_array<double, 3> f_values(boost::extents[5][6][7]); //A boost::multidimensional object boost::detail::multi_array::sub_array<double, 2> sub_f = f_values[2]; //Works boost::multi_array<double, 2> sub_f2 = f_values[2]; //Fails on MSVC10, works on all others } ////////////////// The problem here is with the constructor in multi_array.hpp around line 334. It uses an std::copy, However, if I replace the: std::copy(rhs.begin(),rhs.end(),this->begin()); with the (godawful) direct iteration: auto p_rhs = rhs.begin(); auto p_this = this->begin(); while(p_rhs < rhs.end()) { *p_this = *p_rhs; ++p_rhs; ++p_this; } Things compile fine. Also note, that this same problem with the std::copy happens in pretty much every other constructor/place it is used in this library. (i.e. all over the place, so this 'hack' is pretty useless). It appears to me that the MSVC's STD implementation isn't finding a match a std::copy overload impl with the iterators as: Input iterator as: boost::detail::multi_array::array_iterator<double,const double *,boost::mpl::size_t<2>,boost::detail::multi_array::const_sub_array<double,1>> Output iterator as: boost::detail::multi_array::array_iterator<double,double *,boost::mpl::size_t<2>,boost::detail::multi_array::sub_array<double,1>>, I have attached my test files. Any ideas or patches would be appreciated

2 2

How to specify Boost headers location properly
by Mateusz Loskot 11 Mar '10

11 Mar '10

Hi, I'm trying to learn what is, let's say, canonical form of configuring Boost headers location in Jamfiles of test and example programs for: 1) Libraries that live inside Boost tree (SVN trunk) - Boost libraries. 2) Libraries that are close-to-Boost, it is Sandbox, libraries being reviewed and libraries already reviewed and accepted but not yet included in Boost tree In the case 1), I assume it's required that a library is configured to use headers from the same Boost tree where it is actually included. In the case 2), I assume that it's recommended to configure a library in such way that allows compilation in two ways: a) user copies library to Boost tree and compiles, i.e. examples, by executing bjam inside BOOST_ROOT/libs/LIBRARY/example b) user does not copy library to Boost tree, but keeps it externally and still can do: cd BOOST_ROOT/libs/LIBRARY/example bjam I was explained by folks on #boost channel, that there is no requirement for user to: - set environment variables like BOOST_ROOT - or to set global <include> requirements in user-config.jam, like this: project : requirements <include>/path/to/boost/root ; As I was told, the only entry in user-config.jam related to location of Boost headers should be use-project boost : /path/to/boost/root ; I'm looking at various Jamfiles in trunk and sandbox and I'm a bit confused. For 1st example for the case 1), in trunk/libs/accumulators/example/Jamfile.v2, Boost headers are located by relative path and mysterious $(BOOST_ROOT) exe example : main.cpp : <include>../../.. <include>$(BOOST_ROOT) ; For 2nd example of the case 1), in libs/program_options/example/Jamfile.v2, there is no <include> or <source> requirement configured, so AFAIU, it means global Boost Jamroot takes care of setting headers location. For 1st example for the case 2), it is ITL library that was submitted to review, not yet in Boost tree, I see similar double-include trick in Jamfile of examples: https://svn.boost.org/svn/boost/sandbox/itl/libs/itl/example/Jamfile.v2 exe overlap_counter : overlap_counter_/overlap_counter.cpp : <include>../../.. <include>$(BOOST_ROOT) ; I have seen similar pattern for other libraries in Sandbox. It tells me that either these libraries can be build *only* from within Boost tree or, if user wants to build them from outside Boost tree, it is required to specify the mysterious BOOST_ROOT which in fact should not be required at all [1] [1] http://lists.boost.org/boost-users/2004/09/7755.php It also suggests me that for cases as the one illustrated with Jamfile from the ITL library, if a user doesn't copy a library in to Boost tree or doesn't set BOOST_ROOT or doesn't put the hack of hard-coded Boost headers path as <include> in user-config.jam file, then it is not possible to compile such library (or its examples or tests, if it's headers-only). I started to ask myself: How folks compile libraries while reviewing if these libraries are configured to locate headers with <include>../../.. <include>$(BOOST_ROOT) Does it mean a library submitted to review (or reviewed and accepted but not yet included to Boost) are required to be configured as living in Boost tree? Or, does it mean everybody makes "unofficial" configurations to environment like setting $BOOST_ROOT or hacking user-config.jam. Now, here comes final question derived from all the elaborate introduction: If I assume that a user installs Boost without making any of the not recommended hacks to his environment (let's say, only puts use-project boost directive to user-config.jam), then How should I configure Jamfiles of Boost Geometry (examples and tests) to locate Boost headers in the following two situations: 1) User copies Boost Geometry in to Boost tree and compiles examples and tests 2) User keeps Boost Geometry outside Boost tree and still should be able to compile examples and tests using his Boost installation. By the way, I have started discussing this [2] problem on Boost Geometry mailing list but there are still some confusions [3] that make this issue far from clarified. [2] http://lists.osgeo.org/pipermail/ggl/2010-March/000657.html [3] http://lists.osgeo.org/pipermail/ggl/2010-March/000662.html I'd appreciate if someone here could teach me, once and for good, how to make it right. Best regards, -- Mateusz Loskot, http://mateusz.loskot.net Charter Member of OSGeo, http://osgeo.org

4 12

Participation in google summer of code 2010
by vikas kushwaha 11 Mar '10

11 Mar '10

I would like to participate in gsoc 2010. I was thinking of audio library for boost, as a project. Will it be a good project to work upon. Thanks in advance.

2 1

[Log] Review Results
by Roland Bock 11 Mar '10

11 Mar '10

Hi, here are my review results. I have organized it in two parts. A relatively short one answering the formal review questions and a second part that lists a lot of additional details. Attached is a short program which contains some of the tests I did. - Should the library be accepted? ================================= YES, but some things should be fixed first (see also details below): - Compiler warnings: I get tons of them. Zero should be the goal. - Completeness of documentation: Some things are simply missing and can only be found in the code - Compile time for the library - A lot of details are listed below. Of course, I would like to see all of them addressed, but my vote does not depend on every single one of them. - What is your evaluation of the design? ======================================== Very flexible, very modularized. I have two concerns, both explained in more detail below. a) In some aspects, it is too flexible, I think. This makes it hard to document (which can be seen in the current documentation) and hard to test (the latter is just me guessing). b) It is easy to use it in the wrong way, which leads to exceptions and lost log records, possibly after hours of running, probably in some rare and potentially critical situation. It would be better to somehow prevent the wrong usage at compile time. - What is your evaluation of the implementation? ================================================ I looked at the code only briefly, but what I saw looked good to me (see below). - What is your evaluation of the documentation? =============================================== Very nice examples, very well motivated tutorial. Good overview on details. Was good to read to get an idea of how the library can be used. But (at least for me) it was sometimes tough to combine the information given in examples to the things I want to do myself. Mostly this was because some information is missing (or too well hidden for me). I admit that I got lost on the last few pages, but that was probably due to me not actually experimenting with all of it. See below for specific remarks. - What is your evaluation of the potential usefulness of the library? ===================================================================== Very useful! Logging is one of the core aspects of almost all software. And it is not nearly as easy as it seems at first glance. Thus, it is very useful to have a well designed, well documented, well tested and well supported log library. - Did you try to use the library? With what compiler? Did you have any ====================================================================== problems? ========= I compiled the library (the version from the SVN trunk, because the packaged version from sourceforge did not compile on my system). I also did some experiments to see how easy or hard it was to transfer the information from the documentation to some "real" code. I stumbled over quite a few details, but am convinced that I could use the library, soon. The most annoying problem were the compile warnings I mentioned above. Ubuntu-8.04 64bit, gcc-4.2.4, boost-log from SVN trunk, boost-1.42 - How much effort did you put into your evaluation? A glance? A quick ===================================================================== reading? In-depth study? ======================== I spent several hours reading the documentation and several hours of experimenting (see details below), about 12 hours in total (which includes writing this review). - Are you knowledgeable about the problem domain? ================================================= I wrote the log library which is being used by some 100 components in our company for about two years now. It is based on POCO library (pocoproject.org) Before writing it, I analyzed several log other libraries too, of course. So I would not consider myself an expert, but not really a newbie, either :-) ------------------------------------------------------------------------ MORE DETAILS ------------------------------------------------------------------------ Missing Features: ================= Maybe I overlooked it, but I did not find the following: - Have several sinks share a formatter: The programs in my company typically produce several log files: - debug: Contains everything - warn: Contains everything which is at least a warning - error: Contains everything which is error or fatal Messages in all files are formatted in the same way. I understand that I could create several file sinks and each is given its own formatter. But that would mean that the same formatting is done several times. Is there a way to have several sinks with different filter but sharing the log records? Compiling: ========== - Library Code: As mentioned in an earlier mail, it takes awfully long to compile the library, which is due to one file: formatter_parser.cpp The parsing is based on Boost.Spirit (classic) and takes 20 minutes to compile on my maschine (Ubuntu-8.04 64bit, gcc-4.2.4, Intel Quad Core, 2.5GHz) I don't know how much can be gained by switching to Spirit 2.1. But I am sure a way has to be found to accelerate the compilation. Otherwise, users will do what I did: Get nervous about the long compilation time and tend to press Ctrl-C before the work is done. - My Code: All of the code in our company is compiled with the following settings (gcc-4.2.4): -Wall -Wreorder -Wnon-virtual-dtor -Wno-non-template-friend -Woverloaded-virtual -Wsign-promo -Wextra -fvisibility=hidden We strive for code that compiles without warnings, but with boost.log, I get tons of warnings. What I have seen so far should be easy to get rid of. But I will not be able to use the boost.log library unless the warnings are taken care of. Debugging: ========== - I tried to add a timestamp to each written record. I used core->add_global_attribute("TimeStamp", boost::make_shared< attrs::local_clock >()); backend->set_formatter( fmt::stream << fmt::date_time< std::tm >("TimeStamp") << ": [" << fmt::attr< severity_level >("Severity") << "] " << fmt::message() ); This compiled, but when I ran my program, I got an exception message which did not really tell what was wrong: ----------------- terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::log_mt_posix::missing_value> >' what(): Requested attribute value not found ----------------- I think, at least it should tell which attribute value was not found. But the real reason for the exception was that the types of the attribute and the attribute value did not match. - Is there a way to get to these exceptions earlier? As of now, in the example above, the exception is thrown when a logger is used. This is much too late. I need to initialize logging and then be sure that logging will work for the components that use it (within limits of course, e.g. filesystem being full). But mis-configurations like above must be detected before the real work is done in the components using the log system. For instance when constructing the logger. - Of course, best would be to prevent such stuff at compile time (no idea how, though) Design: ======= - Flexibility: To be honest, in some aspects, I would have preferred to see less flexibility, for two reasons: a) the documentation effort rises and I suspect that the missing completeness of the documentation is partly a result of over-flexibility b) the testing effort rises (I guess). It will be hard to make sure that everything works as expected. An example is log record formatting via Boost.Lamda, Boost.Format, String Templates, Custom-Formatting, which is all shown by example in the tutorial, but the detailed description of Formatters does not even mention String Templates or Custom Formatting. - Easy to misuse: If your formatter requires a severity and the logger does not add one to the log record or adds a severity of the wrong type, using that logger results in an exception, see attached code. It is unlikely that this will happen in some part of the code which is executed frequently, because the developer would certainly notice. But what if someone would add a non-matching logger at some rarely executed but critical code section? Not only would the code fail with an exception, it would also fail to log the message! As of now, I would have to provide some factory method which produces loggers for anyone who needs one. And no logger is ever to be constructed otherwise. Just to make sure that everybody uses a logger that actually produces digestible log records. I can live with that, but I do not consider it a good solution. Code Quality: ============= - Readability: I did not read through all the code, and none in very much detail, but the parts I looked at, looked well organized and readable. - Names: - namespace trivial: I admit that I don't like that, especially since its content is not exactly trivial. Easy, basic or default maybe, I don't know. But not trivial :-) Documentation: ============== Very nice examples, very well motivated tutorial. Good overview on details. Was good to read to get an idea of how the library can be used. But (at least for me) it was sometimes tough to combine the information given in examples to the things I want to do myself. I admit that I got lost on the last few pages, but was probably due to me not actually experimenting with all of it. Some things should be looked into, though: Build process: - it would be nice, if the default build configuration were documented - I wonder which build configuration will be used when the library is included in boost? Tutorial: - Trivial logging with filters: The "Tip" that streaming expressions are not evaluated if the message is filtered should probably be a warning of potential hazard. It is certainly a good feature, but the semantics of the macros are not immediately obvious so I would give the "Tip" a more critical nature. - none of the code examples is complete. They are missing the namespace declarations documented in the introduction, at least. While brevity is good, it would be very helpful to have complete, compiling examples in the tutorial Missing Documentation: - Not all include files are mentioned. This makes it unnecessarily tough to follow the examples. For instance, boost/log/utility/empty_deleter.hpp is required to use logging::empty_deleter. I could not find this information in all of the turorial and details. Another, maybe even more critical example: The BOOST_LOG macro is defined in boost/log/sources/record_ostream.hpp Again I did not find that information in the tutorial. - I understand that I can use my own set of severity levels. It is well documented how to filter messages using these severity levels. But I have no idea how to format messages with these severity levels? I do not want the number, I'd like a string, of course. I looked it up in the trivial.hpp, but it should be part of the documentation. - Maybe I missed it, but I did not find an overview of the log rotation options, including a description of the placeholders in strings. Specific question: Can I tell the log rotation to take place at midnight (regardless of the amount of data which is being logged)? I would like to stress this last item one more time: The examples are excellent, but they should be accompanied by more formal tables which show the complete set of options, e.g. which placeholders can be used in a filename format or a list of public (protected?) member functions of a logger. For instance, I found the logger::strm() member in examples only.

3 6