[thread] Problems with TSS and static thread library
Hi, I've been trying to resolve a problem which seems to be related to TSS and the static build of the Boost.Thread library. I'm using MSVC 8.0 (no SP applied) under WinXP and Boost RC_1_34_0 (CVS) from last week. The problem only manifests itself on one machine and only for release builds. The machine in question is a XP SP2 Pro machine with dual processors. The same code works fine under a XP SP2 Pro SMP with 2 CPUs + HT enabled, and also under OpenSUSE 10.0 + gcc 4.0 (same hardware as the failing machine). What seems to be happening is that the thread_specific_ptr's cleanup method is called twice with a non-null pointer for the main thread of the application. Changing the application to use the DLL variant of the thread library cures the problem (or, at least the application no longer crashes with an access violation). The good news is that the problem is consistent - it always occurs for my test application. The bad news is that the environment is pretty complex, and I can't reproduce the problem with a simple test case. I figured I could attach some output from my test application together with source code excerpts (with tracing hacks added) and hope that somebody could give me a hint on what's actually going on. I added debug symbols and disabled optimization for the release build to try and get better call stack information, but the call stack still seems a bit messed up. Sorry for the longish output, but here goes: ------------------------- excerpt from Context.cpp ------------------------- #ifdef BOOST_MSVC __declspec(thread) static int all = 0, reals = 0; #endif // // STATIC DATA // Context::MapPtrT Context::g_pCtxMap(&Context::Cleanup); void Context::Cleanup(MapT* pCtxMap) { #ifdef BOOST_MSVC all++; DWORD thread = ::GetCurrentThreadId(); std::cout << std::hex << std::setw(8) << thread << ": Context::Cleanup(" << pCtxMap << "), all: " << all << ", reals: " << reals << std::endl; if (pCtxMap) { reals++; std::cout << "Clearing, reals: " << reals << std::endl; assert(reals == 1); pCtxMap->clear(); std::cout << "Deleting" << std::endl; delete pCtxMap; } #else delete pCtxMap; #endif } // // HELPERS // Context::MapT& Context::ContextMap() { if (!g_pCtxMap.get()) { g_pCtxMap.reset(new MapT); } return *(g_pCtxMap.get()); } -------------------- RELEASE BUILD OUTPUT -------------------- testing.unit-test C:\users\cc\builds\checkout\knmc\build\src\libs\oasis\oasistest\msvc-8.0\release\debug-symbols-on\optimization-off \threading-multi\oasistest.passed Running 402 test cases... 2b4: Context::Cleanup(00D22528), all: 1, reals: 0 Clearing, reals: 1 Deleting 7d8: Context::Cleanup(00000000), all: 1, reals: 0 754: Context::Cleanup(00000000), all: 1, reals: 0 bc: Context::Cleanup(00000000), all: 1, reals: 0 1b4: Context::Cleanup(00000000), all: 1, reals: 0 9c: Context::Cleanup(00000000), all: 1, reals: 0 5b8: Context::Cleanup(00000000), all: 1, reals: 0 530: Context::Cleanup(00000000), all: 1, reals: 0 39c: Context::Cleanup(00000000), all: 1, reals: 0 58c: Context::Cleanup(00000000), all: 1, reals: 0 6b8: Context::Cleanup(00000000), all: 1, reals: 0 318: Context::Cleanup(00000000), all: 1, reals: 0 *** No errors detected 134: Context::Cleanup(00D253F8), all: 1, reals: 0 Clearing, reals: 1 Deleting 134: Context::Cleanup(00D24658), all: 2, reals: 1 Clearing, reals: 2 ---------------- Error popup occurs here: ---------------- Unhandled exception at 0x0054ea2a in oasistest.exe: 0xC0000005: Access violation reading location 0x00000025. ------------------ Call stack when entering the debugger: ------------------ oasistest.exe!std::_Tree<std::_Tmap_traits<oasis::utility::detail::ContextKey,std::stack<boost::any,std::deque<boost::any,std::allocator<boost::any>
,std::less<oasis::utility::detail::ContextKey>,std::allocator<std::pair<oasis::utility::detail::ContextKey
const ,std::stack<boost::any,std::deque<boost::any,std::allocator<boost::any> > >
,0> ::_Erase(std::_Tree_nod<std::_Tmap_traits<oasis::utility::detail::ContextKey,std::stack<boost::any,std::deque<boost::any,std::allocator<boost::any>
,std::less<oasis::utility::detail::ContextKey>,std::allocator<std::pair<oasis::utility::detail::ContextKey const ,std::stack<boost::any,std::deque<boost::any,std::allocator<boost::any> > >
,0> >::_Node * _Rootnode=0x00000000) Line 1073 + 0x11 bytes C++ oasistest.exe!std::_Tree<std::_Tmap_traits<oasis::utility::detail::ContextKey,std::stack<boost::any,std::deque<boost::any,std::allocator<boost::any>
,std::less<oasis::utility::detail::ContextKey>,std::allocator<std::pair<oasis::utility::detail::ContextKey const ,std::stack<boost::any,std::deque<boost::any,std::allocator<boost::any> >
,0> ::_Erase(std::_Tree_nod<std::_Tmap_traits<oasis::utility::detail::ContextKey,std::stack<boost::any,std::deque<boost::any,std::allocator<boost::any>
,std::less<oasis::utility::detail::ContextKey>,std::allocator<std::pair<oasis::utility::detail::ContextKey const ,std::stack<boost::any,std::deque<boost::any,std::allocator<boost::any> >
,0> >::_Node * _Rootnode=0x00d24df0) Line 1076 C++ oasistest.exe!std::_Tree<std::_Tmap_traits<oasis::utility::detail::ContextKey,std::stack<boost::any,std::deque<boost::any,std::allocator<boost::any>
,std::less<oasis::utility::detail::ContextKey>,std::allocator<std::pair<oasis::utility::detail::ContextKey const ,std::stack<boost::any,std::deque<boost::any,std::allocator<boost::any> > >
,0> >::clear() Line 955 C++ oasistest.exe!oasis::utility::Context::Cleanup(std::map<oasis::utility::detail::ContextKey,std::stack<boost::any,std::deque<boost::any,std::allocator<boost::any>
,std::less<oasis::utility::detail::ContextKey>,std::allocator<std::pair<oasis::utility::detail::ContextKey const ,std::stack<boost::any,std::deque<boost::any,std::allocator<boost::any> > >
* pCtxMap=[0]()) Line 48 C++ oasistest.exe!boost::detail::function::void_function_invoker1<void (__cdecl*)(boost::shared_ptr<RCF::Session> *),void,boost::shared_ptr<RCF::Session> *>::invoke(boost::detail::function::function_buffer & function_ptr={...}, boost::shared_ptr<RCF::Session> * a0=0x00d24658) Line 114 + 0x7 bytes C++ oasistest.exe!boost::function1<void,boost::shared_ptr<RCF::Session> *,std::allocator<boost::function_base> ::operator()(boost::shared_ptr<RCF::Session> * a0=0x00d24658) Line 692 + 0x15 bytes C++ oasistest.exe!boost::detail::function::void_function_obj_invoker1<boost::detail::tss_adapter<boost::shared_ptr<rsdb::detail::IRsdbApi> ,void,void *>::invoke(boost::detail::function::function_buffer & function_obj_ptr={...}, void * a0=0x00d24658) Line 156 C++ oasistest.exe!boost::function1<void,void *,std::allocator<boost::function_base> >::operator()() + 0x4e bytes C++ oasistest.exe!boost::detail::tss::cleanup() + 0x78 bytes C++ oasistest.exe!boost::thread_specific_ptr<boost::shared_ptr<RCF::Session> ::~thread_specific_ptr<boost::shared_ptr<RCF::Session> >() Line 90 + 0x5e bytes C++ oasistest.exe!`dynamic atexit destructor for 'oasis::utility::Context::g_pCtxMap''() + 0xe bytes C++ msvcr80.dll!doexit(int code=0, int quick=0, int retcaller=0) Line 553 C msvcr80.dll!exit(int code=0) Line 398 + 0xd bytes C oasistest.exe!__tmainCRTStartup() Line 603 C kernel32.dll!7c816fd7() [Frames below may be incorrect and/or missing, no symbols loaded for kernel32.dll] kernel32.dll!7c816fd7()
------------------ DEBUG BUILD OUTPUT ------------------ testing.unit-test C:\users\cc\builds\checkout\knmc\build\src\libs\oa Running 402 test cases... 608: Context::Cleanup(0100E720), all: 1, reals: 0 Clearing, reals: 1 Deleting 1a8: Context::Cleanup(00000000), all: 1, reals: 0 600: Context::Cleanup(00000000), all: 1, reals: 0 128: Context::Cleanup(00000000), all: 1, reals: 0 31c: Context::Cleanup(00000000), all: 1, reals: 0 e4: Context::Cleanup(00000000), all: 1, reals: 0 7d0: Context::Cleanup(00000000), all: 1, reals: 0 168: Context::Cleanup(00000000), all: 1, reals: 0 f8: Context::Cleanup(00000000), all: 1, reals: 0 52c: Context::Cleanup(00000000), all: 1, reals: 0 1a0: Context::Cleanup(00000000), all: 1, reals: 0 784: Context::Cleanup(00000000), all: 1, reals: 0 *** No errors detected 6bc: Context::Cleanup(00FE3838), all: 1, reals: 0 Clearing, reals: 1 Deleting ...updated 5 targets... ----------------------------------- / Johan
Johan Nilsson wrote:
Context::MapPtrT Context::g_pCtxMap(&Context::Cleanup);
What is MapPtrT?
C++ oasistest.exe!boost::detail::tss::cleanup() + 0x78 bytes C++ oasistest.exe!boost::thread_specific_ptr<boost::shared_ptr<RCF::Session>
:~thread_specific_ptr<boost::shared_ptr<RCF::Session> >() Line 90 + 0x5e bytes C++ oasistest.exe!`dynamic atexit destructor for 'oasis::utility::Context::g_pCtxMap''() + 0xe bytes C++ msvcr80.dll!doexit(int code=0, int quick=0, int retcaller=0) Line 553 C msvcr80.dll!exit(int code=0) Line 398 + 0xd bytes C oasistest.exe!__tmainCRTStartup() Line 603 C kernel32.dll!7c816fd7()
It seems that the static destructors are being run after on_thread_exit for the main thread; that's pretty odd. I see some logic in tss.cpp that looks like it should handle the case of ordering/race issues between ~thread_specific_ptr and the thread exit cleanup, but the code is too complicated to tell. Maybe you need to instrument tss.cpp and tss_hooks.cpp to see what is being called and when.
"Peter Dimov" <pdimov@mmltd.net> skrev i meddelandet news:00e301c77b79$c29f7d80$6b01a8c0@pdimov...
Johan Nilsson wrote:
Context::MapPtrT Context::g_pCtxMap(&Context::Cleanup);
What is MapPtrT?
Sorry, should have spelled out the declaration: boost::thread_specific_ptr< std::map< "typeid wrapper", std::stack< boost::any > >
(modulo typos)
C++ oasistest.exe!boost::detail::tss::cleanup() + 0x78 bytes C++ oasistest.exe!boost::thread_specific_ptr<boost::shared_ptr<RCF::Session>
:~thread_specific_ptr<boost::shared_ptr<RCF::Session> >() Line 90 + 0x5e bytes C++ oasistest.exe!`dynamic atexit destructor for 'oasis::utility::Context::g_pCtxMap''() + 0xe bytes C++ msvcr80.dll!doexit(int code=0, int quick=0, int retcaller=0) Line 553 C msvcr80.dll!exit(int code=0) Line 398 + 0xd bytes C oasistest.exe!__tmainCRTStartup() Line 603 C kernel32.dll!7c816fd7()
It seems that the static destructors are being run after on_thread_exit for the main thread; that's pretty odd. I see some logic in tss.cpp that looks
like it should handle the case of ordering/race issues between ~thread_specific_ptr and the thread exit cleanup, but the code is too complicated to tell. Maybe you need to instrument tss.cpp and tss_hooks.cpp to see what is being called and when.
I'm trying to avoid that. Will wait a bit more and cross my fingers for someone to see any "obvious" errors. / Johan
Peter Dimov wrote: [snip]
It seems that the static destructors are being run after on_thread_exit for the main thread; that's pretty odd.
Perhaps this is just due to some implementation details/limitation under the complex TSS implementation for static Boost.Thread under Win32.
I see some logic in tss.cpp that looks like it should handle the case of ordering/race issues between ~thread_specific_ptr and the thread exit cleanup, but the code is too complicated to tell. Maybe you need to instrument tss.cpp and tss_hooks.cpp to see what is being called and when.
I've put in some more time into studying the details of tss(_hooks|_pe|).cpp and I can only agree with you about that the problem should not occur. As I mentioned the call stack looks a bit weird from the release build, which perhaps could indicate memory corruption somewhere else in the program (instead of symbol problems). Wild guess, but I'll still have to dig even deeper into the problem. / Johan
participants (2)
-
Johan Nilsson -
Peter Dimov