[boost] Re: Re: Re: Re: Thread-Local Storage (TLS) and templates

From: Edward Diener (eddielee_at_[hidden])
Date: 2003-02-25 16:47:24

William E. Kempf wrote:
> Edward Diener said:
>> William E. Kempf wrote:
>>>  And it's full of issues.
>>> You are quite limited in what you can safely do within DllMain.  Any
>>> calls to synchronization routines is likely to deadlock the entire
>>> process.
>>
>> I agree that this is so. You can't sit and wait until some other
>> thread has done something, via a Windows synchronization primitive,
>> when you are processing DLL_THREAD_DETACH. What is the situation
>> where this is necessary ?
>
> There are numerous situations where this is necessary.  For example,
> the cleanup mechanism in both Boost.Threads and pthreads-win32 use
> mutexes, which can potentially cause *process* deadlock.  If the TLS
> data is shared across threads, or references data shared across
> threads, or simply calls a routine that does synchronization in the
> cleanup, all of which are not that uncommon, and some of which are
> hard for the programmer to avoid (do you know what routines do
> synchronization internally?), you risk deadlock.

My understanding of TLS data is that it is thread specific and not meant to
be shared across threads. The whole idea is that every thread in a process
gets their own copy of the same data. Why then do you say that "TLS data is
shared across threads, or references data shared across threads" ? The last
issue of doing synchronization in the cleanup I can understand but not that
the data which needs to be synchronized is TLS data itself. I completely
agree with you that there is a serious problem in the DLL_THREAD_DETACH
attempting to do synchronized cleanup, but I am guessing that you may be
using TLS data itself in ways in which it was not intended. I don't believe
TLS data was ever intended as a way to share data between threads but was
intended, as its name implies, to create data that is specific only to a
single thread.

>
>>>  As is calling any routines that load/unload a DLL.
>>
>> The workaround is not to dynamically load/unload a DLL as part of
>> thread processing.
>
> Do you know what routines do this as part of their implementation?  To
> quote the MSDN "Calling imported functions other than those located in
> Kernel32.dll may result in problems that are difficult to diagnose."
> And since a very large number of Win32 API functions are imported...
> I think you see the issue.

I see the issue and although I haven't investigated what Windows API
functions may load/unload a DLL dynamically, something tells me that MS must
publish such a list somewhere so that one knows what to avoid at
DLL_THREAD_DETACH time at least within their own Windows APIs.

>
>> Yes, it is cleaner to do so when one only needs a DLL for a
>> specific time but the overhead of statically linking a DLL into a
>> process instead is minimal, although I agree that dynamic loading is
>> often a cleaner design. I do agree with you that the inability to
>> dynamically load and unload a DLL at
>> DLL_THREAD_ATTACH/DLL_THREAD_DETACH is an unfortunate imposition and
>> that this is poor design on MS's part. I am still not clear whay
>> this is so and why this limitation exists on Windows.
>
> I honestly don't care.  The only time I've ever found this design to
> be unusable is when dealing specifically with the cleanup of TLS
> data, which would be much better implemented as a registered cleanup
> routine in the first place.  Fix this, and I don't care about this
> artifact of the DLL system on Win32 platforms.

OK, given that a registered cleanup routine would not have the restrictions
which DLL_THREAD_DETACH has.

I still don't think it is a TLS issue but rather a thread cleanup issue and
the restrictions imposed by MS's design of that situation. So I can well
understand your chagrin at the tricks you must do in order to cleanup
internal thread data when a thread exits under Windows.

>
>>>  There's
>>> also the issue of forcing the use of a DLL with this scheme, which
>>> many users rightfully dislike (this is why there are so many thread
>>> creation routines on Windows).
>>
>> I could be mistaken but I believe that TLS works just as effectively
>> in static LIBs as it does with DLLs. The difference is that one must
>> do manual initialization routines and finalization routines of TLS
>> data for different threads, as opposed to what one may do
>> automatically using DLL_THREAD_ATTACH/DLL_THREAD_DETACH. But
>> certainly one is not forced to use only DLLs or only static LIBs if
>> the implementation supports both.
>
> Initialization isn't really an issue, as you can do lazy
> initialization (synchronization issues aside, as they are solvable).
> It's the finalization that's an issue, and it's resulted in the
> numerous thread creation routines and the rules for when to use which
> one.  If you call any C RTL routines (which may allocate TLS) you
> can't call CreateThread, but must instead call _beginthread(ex).
> Likewise, if you call any MFC routines you can't call CreateThread or
> _beginthread(ex) but instead must call AfxBeginThread, lest you leak
> TLS data allocated by these routines. This is an issue for
> Boost.Threads, which has it's own thread creation routines, because I
> don't know how a user will use the thread.  It's a problem for other
> libraries if they have thread creation routines, or have user
> registered callbacks executed by the threads created by the library.
> It's a problem if you want to make use of the built in Win32 thread
> pooling mechanisms, since those threads will be created using
> CreateThread.  And so on.

I can see it is a nest of problems because of the
DLL_THREAD_ATTACH/DLL_THREAD_DETACH limitations.

>
>>>>> I won't be as critical as Alexander, but I will agree that the MS
>>>>> TLS implementation has serious design issues which need to be
>>>>> corrected.
>>>>
>>>> OK, this isn't the place to debate Windows TLS, but I have not run
>>>> into such design issues myself.
>>>
>>> You have, you just weren't necessarily aware of it ;).
>>
>> No, I haven't for what I was doing <g> but that doesn't mean you or
>> others haven't. It seems that the problems are MS's design of what
>> one can do in DLL_THREAD_ATTACH/DLL_THREAD_DETACH itself instead of
>> in TLS. If you can point out specific problems in Boost.Threads I
>> would be glad to hear about it.
>
> Problems in Boost.Threads:
>
> 1) We must use a shared library design on Win32, which is something
> several users (rightfully) have problems with, but this is the only
> way to implement thread_specific_ptr<>. (Well, I could also use the
> "solution" MS has used, i.e. you must use Boost.Threads thread
> creation routines if a thread can ever call a routine that uses TLS
> allocated by Boost.Threads, but this is fragile and not very usable
> in a lot of situations today, and will be even more so for every
> thread creation routine added this way.)

As hard as it may be I think you must also support static LIBs along with
DLLs ( shared libraries ). Some users are against distributing DLLs with
their executable and build it with everything statically linked to the RTL.
Mixing static and dynamic RTL routines I have found a recipe for disaster
although others have found that it may work.

>
> 2) User cleanup routines can't safely use any synchronization
> primitives.
>
> 3) User cleanup routines can't safely use any imported functions from
> any DLL other than kernel32.dll.
>
> 4) 2 & 3 can be done with out the knowledge of the user in many
> circumstances.
>
> 5) The DLL from (1) must be statically linked.  Not necessarily a real
> issue, but something unusual that the users must be aware of.

The user issues, unfortunate as they are, must be documented and accepted by
the user if working in Windows. I am not defending MS, since much of this is
their own too simplistic and limiting design surrounding thread cleanup.

My purpose in answering this is not to disagree with you about the
limitations of thread cleanup since I agree with you on nearly all counts.
If I am defending the Windows TLS API it is because I believe that within
the limits of what thread-specific data was supposed to be when the API was
designed, it does its job effectively. But I am gathering that the internal
design for Boost.Threads is to share certain thread-specific data between
threads using synchronization. And I don't think that TLS is much good for
that because of the weakness in thread cleanup under Windows regarding
calling synchronization routines.

A better design for Windows, whether using a callback or DLL_THREAD_DETACH,
would have been to allow the code to wait on synchronization if necessary to
cleanup data effectively. This would allow other thread to keep running
while one cleans up a particular thread. Unfortunately what I gather from
Windows doc is that DllMain and its underlying code was not made re-entrant
to allow this.