Re: [boost] Re: Re: Re: Re: Thread-Local Storage (TLS) and templates

From: William E. Kempf (wekempf_at_[hidden])
Date: 2003-02-25 17:29:17

Edward Diener said:
> William E. Kempf wrote:
>> Edward Diener said:
>>> William E. Kempf wrote:
>>>>  And it's full of issues.
>>>> You are quite limited in what you can safely do within DllMain.  Any
>>>> calls to synchronization routines is likely to deadlock the entire
>>>> process.
>>>
>>> I agree that this is so. You can't sit and wait until some other
>>> thread has done something, via a Windows synchronization primitive,
>>> when you are processing DLL_THREAD_DETACH. What is the situation
>>> where this is necessary ?
>>
>> There are numerous situations where this is necessary.  For example,
>> the cleanup mechanism in both Boost.Threads and pthreads-win32 use
>> mutexes, which can potentially cause *process* deadlock.  If the TLS
>> data is shared across threads, or references data shared across
>> threads, or simply calls a routine that does synchronization in the
>> cleanup, all of which are not that uncommon, and some of which are
>> hard for the programmer to avoid (do you know what routines do
>> synchronization internally?), you risk deadlock.
>
> My understanding of TLS data is that it is thread specific and not meant
> to be shared across threads.

That's the most common use, but it's not mandated by anything.  However,
sharing the TLS data is the less common of the cases I gave.

> The whole idea is that every thread in a
> process gets their own copy of the same data. Why then do you say that
> "TLS data is shared across threads, or references data shared across
> threads" ? The last issue of doing synchronization in the cleanup I can
> understand but not that the data which needs to be synchronized is TLS
> data itself. I completely agree with you that there is a serious problem
> in the DLL_THREAD_DETACH attempting to do synchronized cleanup, but I am
> guessing that you may be using TLS data itself in ways in which it was
> not intended. I don't believe TLS data was ever intended as a way to
> share data between threads but was intended, as its name implies, to
> create data that is specific only to a single thread.

An example of the former, and I believe a valid example, exists in the
next revision to Boost.Threads.  The thread representation is sharable,
i.e. boost::thread uses a ref-counted pimpl idiom to make it copyable and
assignable.  The implementation holds some state information that's
specific to the thread, such as it's running state.  The default
constructor and/or thread::self() needs to be able to access this shared
state, which means it must be contained in a TLS slot.  There you have TLS
data that's shared across threads.  More importantly, this data must be
cleaned up at thread exit by decrementing the ref-count (which has to be
synchronized) and if the ref-count goes to zero, by actually deleting the
data.

But again, the latter is the more likely case, and is likely to occur
quite frequently in MT C++ code.

>>>>  As is calling any routines that load/unload a DLL.
>>>
>>> The workaround is not to dynamically load/unload a DLL as part of
>>> thread processing.
>>
>> Do you know what routines do this as part of their implementation?  To
>> quote the MSDN "Calling imported functions other than those located in
>> Kernel32.dll may result in problems that are difficult to diagnose."
>> And since a very large number of Win32 API functions are imported... I
>> think you see the issue.
>
> I see the issue and although I haven't investigated what Windows API
> functions may load/unload a DLL dynamically, something tells me that MS
> must publish such a list somewhere so that one knows what to avoid at
> DLL_THREAD_DETACH time at least within their own Windows APIs.

*chuckles*  Sorry, there's no such list.  The MSDN pretty much says the
only routines you can trust are those in kernel32.dll.  But even if MS did
publish such a list, do you think third party vendors do?  Have you
ensured you document which of your own routines call such routines?

>>> Yes, it is cleaner to do so when one only needs a DLL for a
>>> specific time but the overhead of statically linking a DLL into a
>>> process instead is minimal, although I agree that dynamic loading is
>>> often a cleaner design. I do agree with you that the inability to
>>> dynamically load and unload a DLL at
>>> DLL_THREAD_ATTACH/DLL_THREAD_DETACH is an unfortunate imposition and
>>> that this is poor design on MS's part. I am still not clear whay this
>>> is so and why this limitation exists on Windows.
>>
>> I honestly don't care.  The only time I've ever found this design to
>> be unusable is when dealing specifically with the cleanup of TLS data,
>> which would be much better implemented as a registered cleanup routine
>> in the first place.  Fix this, and I don't care about this artifact of
>> the DLL system on Win32 platforms.
>
> OK, given that a registered cleanup routine would not have the
> restrictions which DLL_THREAD_DETACH has.
>
> I still don't think it is a TLS issue but rather a thread cleanup issue
> and the restrictions imposed by MS's design of that situation. So I can
> well understand your chagrin at the tricks you must do in order to
> cleanup internal thread data when a thread exits under Windows.

That's a minor fine hair your splitting.  What's the difference between "a
TLS issue" and a "thread cleanup issue" of TLS data?  And if you look
back, I said I wouldn't call it "broken", just that the "implementation
has serious design issues".  So it looks like we're in agreement now.

Want to join in my mini campaign to convince MS to fix this one?

>>>>>> I won't be as critical as Alexander, but I will agree that the MS
>>>>>> TLS implementation has serious design issues which need to be
>>>>>> corrected.
>>>>>
>>>>> OK, this isn't the place to debate Windows TLS, but I have not run
>>>>> into such design issues myself.
>>>>
>>>> You have, you just weren't necessarily aware of it ;).
>>>
>>> No, I haven't for what I was doing <g> but that doesn't mean you or
>>> others haven't. It seems that the problems are MS's design of what
>>> one can do in DLL_THREAD_ATTACH/DLL_THREAD_DETACH itself instead of
>>> in TLS. If you can point out specific problems in Boost.Threads I
>>> would be glad to hear about it.
>>
>> Problems in Boost.Threads:
>>
>> 1) We must use a shared library design on Win32, which is something
>> several users (rightfully) have problems with, but this is the only
>> way to implement thread_specific_ptr<>. (Well, I could also use the
>> "solution" MS has used, i.e. you must use Boost.Threads thread
>> creation routines if a thread can ever call a routine that uses TLS
>> allocated by Boost.Threads, but this is fragile and not very usable in
>> a lot of situations today, and will be even more so for every thread
>> creation routine added this way.)
>
> As hard as it may be I think you must also support static LIBs along
> with DLLs ( shared libraries ). Some users are against distributing DLLs
> with their executable and build it with everything statically linked to
> the RTL. Mixing static and dynamic RTL routines I have found a recipe
> for disaster although others have found that it may work.

I agree totally.  But I've yet to find a way to implement TLS cleanup in a
static lib.  I've been over this again and again with several people who
thought they had a solution, and in the end they've agreed with me on this
one.  I've also been in contact with several key players at MS, and they
don't have any answers either.  I'm of the belief the only solution is for
MS to fix things here.

>> 2) User cleanup routines can't safely use any synchronization
>> primitives.
>>
>> 3) User cleanup routines can't safely use any imported functions from
>> any DLL other than kernel32.dll.
>>
>> 4) 2 & 3 can be done with out the knowledge of the user in many
>> circumstances.
>>
>> 5) The DLL from (1) must be statically linked.  Not necessarily a real
>> issue, but something unusual that the users must be aware of.
>
> The user issues, unfortunate as they are, must be documented and
> accepted by the user if working in Windows. I am not defending MS, since
> much of this is their own too simplistic and limiting design surrounding
> thread cleanup.

One of the problems is that right now I can't even gaurantee the
Boost.Threads implementation doesn't violate 2&3 in ways that won't
someday cause issue.  For instance, I must call delete, and the memory
allocation routines have to be doing some sort of synchronization, so in
theory at least, I must violate 2.  I doubt this will ever cause a
deadlock, but...

> My purpose in answering this is not to disagree with you about the
> limitations of thread cleanup since I agree with you on nearly all
> counts. If I am defending the Windows TLS API it is because I believe
> that within the limits of what thread-specific data was supposed to be
> when the API was designed, it does its job effectively. But I am
> gathering that the internal design for Boost.Threads is to share certain
> thread-specific data between threads using synchronization. And I don't
> think that TLS is much good for that because of the weakness in thread
> cleanup under Windows regarding calling synchronization routines.

Not just sharing TLS data, but TLS data referencing shared data as well.

But again, the biggest issue of all is the restriction on calling routines
that load/unload DLLs.

-- 
William E. Kempf