Re: TSAN SEGFAULT in Boost.Beast CI

From: William Linkmeyer (wlink10_at_[hidden])
Date: 2022-08-13 15:08:56

(For example, with the same minimal sysroot, compiler and stdlib, tsan may pass on Ubuntu 18.x and fail on 20.x.)

WL

> On Aug 13, 2022, at 11:06 AM, William Linkmeyer <wlink10_at_[hidden]> wrote:
> 
> If these CI tests are running under a container, could you please link to that containerfile? I have noticed that clang’s tsan in particular behaves differently on different operating systems, even with the same compiler, sysroots and stdlib. 
> 
> WL
> 
>>> On Aug 13, 2022, at 10:53 AM, Richard Hodges <hodges.r_at_[hidden]> wrote:
>>> 
>> 
>> 
>> 
>>> On Sat, 13 Aug 2022 at 16:48, William Linkmeyer <wlink10_at_[hidden]> wrote:
>>> That’s interesting. I don’t see any reason why that lock would fail. Clang’s analyzers do come up with false positives sometimes. 
>>> 
>>> If I can get around to it this weekend, I’ll run this under valgrind. If that comes back positive, I’ll run it under a couple different compilers and their default standard libraries; circle, clang, apple clang, msvc. (This tsan looks like it’s using gcc‘s non-extended stdlib.)
>> 
>> Thank you William. Your support is greatly appreciated.
>> For what it's worth, our CI does run tests under valgrind, and they pass.
>> An example run can be found here: https://drone.cpp.al/boostorg/beast/326/11/2
>> 
>>  
>>> 
>>> But it’ll be a busy weekend, I’m trying to learn Haskell and put together a pet project. So, if anyone has the time, the notes above would be what I’d do. 
>>> 
>>> WL
>>> 
>>> > On Aug 12, 2022, at 2:25 PM, Richard Hodges via Boost <boost_at_[hidden]> wrote:
>>> > 
>>> > Dear all,
>>> > 
>>> > You may have noticed that one of the Beast CI checks fails on the master
>>> > and develop branches.
>>> > 
>>> > The failure is a segfault in thread sanitizer. Running on my local machine
>>> > I get the stack trace embedded below.
>>> > 
>>> > This does not happen if Asio is reverted back to version 1.79.0
>>> > 
>>> > Chris has confirmed that the change in Asio that caused this is in
>>> > asio::spawn.
>>> > 
>>> > Note that at this stage there is no reason to suspect that Asio is actually
>>> > at fault. I believe it to be a false positive. However, I am an absolute
>>> > noob when it comes to the Dark Arts of thread sanitizer. I have no idea
>>> > what to blacklist.
>>> > 
>>> > This email has two aims:
>>> > 
>>> > 1. To prevent multiple people wasting their time trying to get to the
>>> > bottom of this,
>>> > 2. To encourage some sage TSAN-maester to emerge who may guide me to
>>> > enlightenment.
>>> > 
>>> > Here is the error and stack trace:
>>> > 
>>> > testing.capture-output
>>> > bin.v2/libs/beast/test/beast/http/read.test/gcc-12/debug/link-static/thread-sanitizer-norecover/threading-multi/visibility-hidden/read.run
>>> > ====== BEGIN OUTPUT ======
>>> > beast.http.read
>>> > ThreadSanitizer:DEADLYSIGNAL
>>> > ==1761540==ERROR: ThreadSanitizer: SEGV on unknown address
>>> > 0x7f6ea78ff000 (pc 0x7f6eaaaba0d0 bp 0x000000000000 sp 0x7f6ea783d910
>>> > T1761542)
>>> > ==1761540==The signal is caused by a READ memory access.
>>> >    #0 __sanitizer::StackDepotBase<__sanitizer::StackDepotNode, 1,
>>> > 20>::Put(__sanitizer::StackTrace, bool*) <null> (libtsan.so.2+0xba0d0)
>>> >    #1 __tsan::CurrentStackId(__tsan::ThreadState*, unsigned long)
>>> > <null> (libtsan.so.2+0x8c48f)
>>> >    #2 __sanitizer::DD::MutexInit(__sanitizer::DDCallback*,
>>> > __sanitizer::DDMutex*) <null> (libtsan.so.2+0xac534)
>>> >    #3 __tsan::DDMutexInit(__tsan::ThreadState*, unsigned long,
>>> > __tsan::SyncVar*) <null> (libtsan.so.2+0x9a3f8)
>>> >    #4 __tsan::MetaMap::GetSync(__tsan::ThreadState*, unsigned long,
>>> > unsigned long, bool, bool) <null> (libtsan.so.2+0xa85dc)
>>> >    #5 __tsan_atomic32_fetch_add <null> (libtsan.so.2+0x783e9)
>>> >    #6 __gnu_cxx::__exchange_and_add(int volatile*, int)
>>> > /usr/include/c++/12/ext/atomicity.h:66 (read+0x41ad46)
>>> >    #7 __gnu_cxx::__exchange_and_add_dispatch(int*, int)
>>> > /usr/include/c++/12/ext/atomicity.h:101 (read+0x41ad46)
>>> >    #8 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release_last_use()
>>> > /usr/include/c++/12/bits/shared_ptr_base.h:187 (read+0x41ad46)
>>> >    #9 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release()
>>> > /usr/include/c++/12/bits/shared_ptr_base.h:361 (read+0x40d87c)
>>> >    #10 std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count()
>>> > /usr/include/c++/12/bits/shared_ptr_base.h:1071 (read+0x41b43c)
>>> >    #11 std::__shared_ptr<boost::asio::detail::strand_executor_service::strand_impl,
>>> > (__gnu_cxx::_Lock_policy)2>::~__shared_ptr()
>>> > /usr/include/c++/12/bits/shared_ptr_base.h:1524 (read+0x424db7)
>>> >    #12 std::shared_ptr<boost::asio::detail::strand_executor_service::strand_impl>::~shared_ptr()
>>> > /usr/include/c++/12/bits/shared_ptr.h:175 (read+0x424de3)
>>> >    #13 boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>,
>>> > 0ul> const, void>::~invoker() <null> (read+0x486f4d)
>>> >    #14 boost::asio::detail::executor_op<boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>,
>>> > 0ul> const, void>, boost::asio::detail::recycling_allocator<void,
>>> > boost::asio::detail::thread_info_base::default_tag>,
>>> > boost::asio::detail::scheduler_operation>::do_complete(void*,
>>> > boost::asio::detail::scheduler_operation*, boost::system::error_code
>>> > const&, unsigned long) <null> (read+0x49ff36)
>>> >    #15 boost::asio::detail::scheduler_operation::complete(void*,
>>> > boost::system::error_code const&, unsigned long)
>>> > boost/asio/detail/scheduler_operation.hpp:40 (read+0x513742)
>>> >    #16 boost::asio::detail::scheduler::do_run_one(boost::asio::detail::conditionally_enabled_mutex::scoped_lock&,
>>> > boost::asio::detail::scheduler_thread_info&, boost::system::error_code
>>> > const&) boost/asio/detail/impl/scheduler.ipp:492 (read+0x501be9)
>>> >    #17 boost::asio::detail::scheduler::run(boost::system::error_code&)
>>> > boost/asio/detail/impl/scheduler.ipp:210 (read+0x5008af)
>>> >    #18 boost::asio::io_context::run()
>>> > boost/asio/impl/io_context.ipp:63 (read+0x4f54d6)
>>> >    #19 boost::beast::test::enable_yield_to::enable_yield_to(unsigned
>>> > long)::{lambda()#1}::operator()() const <null> (read+0x41480d)
>>> >    #20 void std::__invoke_impl<void,
>>> > boost::beast::test::enable_yield_to::enable_yield_to(unsigned
>>> > long)::{lambda()#1}>(std::__invoke_other,
>>> > boost::beast::test::enable_yield_to::enable_yield_to(unsigned
>>> > long)::{lambda()#1}&&) <null> (read+0x4bb4bc)
>>> >    #21 std::__invoke_result<boost::beast::test::enable_yield_to::enable_yield_to(unsigned
>>> > long)::{lambda()#1}>::type
>>> > std::__invoke<boost::beast::test::enable_yield_to::enable_yield_to(unsigned
>>> > long)::{lambda()#1}>(boost::beast::test::enable_yield_to::enable_yield_to(unsigned
>>> > long)::{lambda()#1}&&) <null> (read+0x4b8c78)
>>> >    #22 void std::thread::_Invoker<std::tuple<boost::beast::test::enable_yield_to::enable_yield_to(unsigned
>>> > long)::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) <null>
>>> > (read+0x4b5b00)
>>> >    #23 std::thread::_Invoker<std::tuple<boost::beast::test::enable_yield_to::enable_yield_to(unsigned
>>> > long)::{lambda()#1}> >::operator()() <null> (read+0x4b1f74)
>>> >    #24 std::thread::_State_impl<std::thread::_Invoker<std::tuple<boost::beast::test::enable_yield_to::enable_yield_to(unsigned
>>> > long)::{lambda()#1}> > >::_M_run() <null> (read+0x4ab02e)
>>> >    #25 execute_native_thread_routine <null> (libstdc++.so.6+0xdbb72)
>>> >    #26 __tsan_thread_start_func <null> (libtsan.so.2+0x393ef)
>>> >    #27 start_thread <null> (libc.so.6+0x8ce2c)
>>> >    #28 clone3 <null> (libc.so.6+0x1121af)
>>> > 
>>> > ThreadSanitizer can not provide additional info.
>>> > SUMMARY: ThreadSanitizer: SEGV (/lib64/libtsan.so.2+0xba0d0) in
>>> > __sanitizer::StackDepotBase<__sanitizer::StackDepotNode, 1,
>>> > 20>::Put(__sanitizer::StackTrace, bool*)
>>> > ==1761540==ABORTING
>>> > 
>>> > EXIT STATUS: 66
>>> > ====== END OUTPUT ======
>>> > 
>>> > 
>>> > 
>>> > 
>>> > -- 
>>> > Richard Hodges
>>> > hodges.r_at_[hidden]
>>> > office: [phone]
>>> > home: [phone]
>>> > mobile: [phone]
>>> > 
>>> > _______________________________________________
>>> > Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost