Re: [boost] [log] Review-ready version in the Vault

Subject: Re: [boost] [log] Review-ready version in the Vault
From: vicente.botet (vicente.botet_at_[hidden])
Date: 2009-02-11 12:32:43

----- Original Message ----- 
From: "vicente.botet" <vicente.botet_at_[hidden]>
To: <boost_at_[hidden]>
Sent: Wednesday, February 11, 2009 4:46 PM
Subject: Re: [boost] [log] Review-ready version in the Vault

> 
> ----- Original Message ----- 
> From: "vicente.botet" <vicente.botet_at_[hidden]>
> To: <boost_at_[hidden]>
> Sent: Wednesday, February 11, 2009 11:28 AM
> Subject: Re: [boost] [log] Review-ready version in the Vault
> 
> 
>> I have some innocent questions: 
>> * Is logging thread-safe? 
>> * If yes, are the lines interleaved as it is the case for output streams? 
>> * If not could you point out how and where in the implementation this is handled?
> 
> Hi again,
> 
> well i have found some anwers to my questions on the document. I'll come back later on.
> 

Oh, I forget, Bravo! Excellent documentation.

Reading the code I see a lot of 

    try{ 
        //
    } catch (...){ 
        // Something has gone wrong. As the library should impose minimum influence
        // on the user's code, we simply mimic here that the record is not needed.
    }

Does it mean that the application is unable to know when things go wrong?

I'm mainly interested on how the library behaves on the context of multi threaded programs, performances, ...

A trivial question, are all the sources associated to all the sinks? If this is the case, how the core is made thread-safe, a single mutex, Which operations will occur more often read-only or write? It is worth to use a shared_mutex?
Why don't let the user the ability of instantiating cores?

If I have understood on multi threaded applications we don't need to always use logger_mt but we need to use synchronous_sink or asynchronous_sink.

The single difference between synchronous_sink and asynchronous_sink is that the later will do the formating and writing later on on an other thread, isn't it? In both cases there is a mutex associated to the sink. This mutex could be the bottleneck of the system, only one mutex resource for all the user threads.

Another approach for the asynchronous_sink could be to use a queue of log records for each thread (thread specific storage). Each log records must be timestamped with the creation date and a monotonic counter (time is not enough fine grained) and as the queue is specific to the thread no need to use a mutex.  
On the other side there is a concentrator which takes one by one the elements ordered by the timestamp+counter.

So only the current thread can push on this queue because is specific to the thread. There is a single thread, the concentrator that pops from these queue. In this context we can ensure thread safety without locking as far as the queue has at least two messages. (The Fork/Join Java framework and Boost.ThreadPool use this technique). 

There is one more issue here, because all the threads can log, while on the ThreadPool only the threads in the pool can add tasks on the queue of the worker thread. So the concentrator needs to have access to all the thread_specific storage. 

The Boost.Interthreads library defines a thread_specific_shared_ptr class which extends the thread_specific_ptr class with synchronized access to thread_specific_shared_ptr of a thread from another, either giving the thread::id or iterating on a  map : thread::id  -> shared_ptr<stored_data>.

I have not done yet any performances comparation, but I think that the bottleneck is avoided.

In addition this will solve the issue with log records that are weakly ordered in a multithreaded applications.

If I have time before the review I'll try to use this approach to define an strict_asynchronous_sink which will ensure strict order and minimize the contention. Have you some performance test I can use to compare?

Thanks,
Vicente