Re: [Boost-users] Threads, condition variables, and signals

From: Scott Gifford (sgifford_at_[hidden])
Date: 2007-10-08 14:27:53

Thanks for your help, Zeljko!  A few questions below...

Zeljko Vrba <zvrba_at_[hidden]> writes:

> On Wed, Oct 03, 2007 at 07:47:57PM -0400, Scott Gifford wrote:
>> 
>> As with most Unix servers, the server is shut down with an OS signal
>> telling the server to shut down.  The server catches this signal, then
>> simply calls exit(1).  Any important cleanup happens in the
>> destructors of global variables, so it will happen properly no matter
>> how the server exits.
>> 
> This is not "properly"; I guessed your problem upfront before I had
> finished reading this very paragraph :)
>
> First, you need to block SIGINT in all threads except the main thread.
> If you don't have a main thread, make it just for the purpose of signal
> handling.  Otherwise, SIGINT will be delivered to a random thread.
> Maybe that's OK in your case.

It's not OK in my case, but according to pthread_signal(3), Linux
threads will do the right thing accidentally (each thread has its own
PID, so if I signal the PID the server had when it started, that will
always be the first thread).  Still, the point of using boost::thread
is portability, so I should probably figure out something more robust.

I don't see anything in boost::thread to handle thread signal masks.
Does anybody know of a portable way to handle this, or will I just
have to write it for pthreads and adapt to other environments as I
port my application?

> Second, you don't need to lock the mutex before signaling the condition
> variable (this is what causes your deadlock).

I was under the impression it was necessary to avoid deadlocks.  

For example, if I have code like this:

    // Global
    bool flag;
    boost::mutex flagMutex;
    boost::condition flagCond;

    // Thread 1
    {
      boost::mutex::scoped_lock 
        lock(flagMutex);
      while (!flag)
        flagCond->wait(lock);
    }

    // Thread 2
    flag = true;
    flagCond->notify_all();

If flag is initially false, I could have a flow like this:
   
    Thread 1                      Thread 2
    {
      boost::mutex::scoped_lock 
        lock(flagMutex);
      while(!flag)
                                  flag = true;
                                  flagCond->notify_all();
        flagCond->wait(lock);

Basically: Thread 1 checks the condition before it is set, then Thread
2 notifies of a change before Thread 1 starts waiting, then Thread 1
starts waiting for a change, but it will never see one, so it hangs
forever.

If Thread 2 acquires the mutex before changing the flag, this race
condition can't happen, because it cannot change flag between Thread
1's test and its wait.

Is there some mechanism I'm not aware of that prevents this race from
happening?

> Third, how do you clean up threads which are NOT waiting at the condition
> variable at the moment of signal arrival?

They will notice the flag has changed when they are done with their
work, and since their work is in fairly small units, they will always
finish within a few seconds, which is fine.

> Fourth, your cleanup is NOT "proper" in any way because your cleanup
> executes in the context of signal handler.  In that context (i.e.
> before the signal handler returns - which is never in your case), only
> async-signal safe functions may be used.  Neither mutex locking nor
> condition signal is async signal safe.

Ah, this is where I was confused.  For some reason, I thought that
exit(3) was safe to call from a signal handler, and it would leave the
signal handling context and clean up safely, but I'm not sure where I
got that idea.

> So how do you do it "properly." Hm: have another, 'main' thread just for
> the purposes of signal handling; have SIGINT unblocked in this thread and
> block it in all other threads.  That thread does something like:
>
> while(!flag)
>   sigsuspend(...);
>
> and SIGINT just sets the flag to true.  When the while() loop exits,
> you're out of the signal context and then you may use functions such
> as pthread_cancel() to cancel all threads, or pthread_kill() to
> explicitly deliver signal to all threads and yes, also
> pthread_cond_signal() and pthread_cond_broadcast().  Then use
> pthread_join() to wait that all threads finish and _then_ call exit
> from the main thread.  (If you use pthread_kill, you have the same
> caveat with async signal safe functions).

Thanks, I'll try that!

[...]

> Also, I don't see how you can use them to notify threads that also do
> some work outside of their monitor, i.e. if they are structured like:
>
> while(1) {
>  // get mutex
>  // wait on condition
>  // do some work
>  // release mutex
>  // do some more work (*)
> }
>
> If you signal the variable while the thread is executing in (*), it
> will never pick up the signal.

The work will always finish within a second or so, and it checks the
condition before waiting on the condition variable, as in the example
code above.
 
> How do you do it in Boost.Thread - I don't know.  The whole idea of using
> destructors to clean up global data in a multi-threaded program sounds
> like calling for trouble.  While the destructors themselves may use locks,
> in how many threads is the destructor list walked and cleaned (the code
> generated by the compiler to walk over the list of global objects and call
> destructor for each) executing?  Does your runtime library and/or compiler
> guarantee that every global destructor is executed exactly once even in a
> MT setting?  (This sounds kinda the inverse of the threadsafe singleton
> pattern.)

I have no idea how I would go about looking for this guarantee, but it
seems that if an environment doesn't provide it, it would be
completely impossible to use global data reliably, making it too
broken to use.  I'm using g++ 4.1.2.  Any pointers as to where to look
for some sort of guarantee like this?

>> Does anybody have any suggestions for a straightforward way to handle
>> this properly?
>> 
> Um, signals and threads don't mix well.  No "straightforward" solution.
> Read about and understand async-signal safety.

Blech, that's too bad.  I should have known this would not be easy.

----Scott.