Re: TSAN SEGFAULT in Boost.Beast CI

From: William Linkmeyer (wlink10_at_[hidden])
Date: 2022-08-13 15:11:49

(And fails in a very similar way to this, in my experience; otherwise stable behavior, segfault, abort.)

WL

> On Aug 13, 2022, at 11:08 AM, William Linkmeyer <wlink10_at_[hidden]> wrote:
> 
> (For example, with the same minimal sysroot, compiler and stdlib, tsan may pass on Ubuntu 18.x and fail on 20.x.)
> 
> WL
> 
>>> On Aug 13, 2022, at 11:06 AM, William Linkmeyer <wlink10_at_[hidden]> wrote:
>>> 
>> If these CI tests are running under a container, could you please link to that containerfile? I have noticed that clang’s tsan in particular behaves differently on different operating systems, even with the same compiler, sysroots and stdlib. 
>> 
>> WL
>> 
>>>> On Aug 13, 2022, at 10:53 AM, Richard Hodges <hodges.r_at_[hidden]> wrote:
>>>> 
>>> 
>>> 
>>> 
>>>> On Sat, 13 Aug 2022 at 16:48, William Linkmeyer <wlink10_at_[hidden]> wrote:
>>>> That’s interesting. I don’t see any reason why that lock would fail. Clang’s analyzers do come up with false positives sometimes. 
>>>> 
>>>> If I can get around to it this weekend, I’ll run this under valgrind. If that comes back positive, I’ll run it under a couple different compilers and their default standard libraries; circle, clang, apple clang, msvc. (This tsan looks like it’s using gcc‘s non-extended stdlib.)
>>> 
>>> Thank you William. Your support is greatly appreciated.
>>> For what it's worth, our CI does run tests under valgrind, and they pass.
>>> An example run can be found here: https://drone.cpp.al/boostorg/beast/326/11/2
>>> 
>>>  
>>>> 
>>>> But it’ll be a busy weekend, I’m trying to learn Haskell and put together a pet project. So, if anyone has the time, the notes above would be what I’d do. 
>>>> 
>>>> WL
>>>> 
>>>> > On Aug 12, 2022, at 2:25 PM, Richard Hodges via Boost <boost_at_[hidden]> wrote:
>>>> > 
>>>> > Dear all,
>>>> > 
>>>> > You may have noticed that one of the Beast CI checks fails on the master
>>>> > and develop branches.
>>>> > 
>>>> > The failure is a segfault in thread sanitizer. Running on my local machine
>>>> > I get the stack trace embedded below.
>>>> > 
>>>> > This does not happen if Asio is reverted back to version 1.79.0
>>>> > 
>>>> > Chris has confirmed that the change in Asio that caused this is in
>>>> > asio::spawn.
>>>> > 
>>>> > Note that at this stage there is no reason to suspect that Asio is actually
>>>> > at fault. I believe it to be a false positive. However, I am an absolute
>>>> > noob when it comes to the Dark Arts of thread sanitizer. I have no idea
>>>> > what to blacklist.
>>>> > 
>>>> > This email has two aims:
>>>> > 
>>>> > 1. To prevent multiple people wasting their time trying to get to the
>>>> > bottom of this,
>>>> > 2. To encourage some sage TSAN-maester to emerge who may guide me to
>>>> > enlightenment.
>>>> > 
>>>> > Here is the error and stack trace:
>>>> > 
>>>> > testing.capture-output
>>>> > bin.v2/libs/beast/test/beast/http/read.test/gcc-12/debug/link-static/thread-sanitizer-norecover/threading-multi/visibility-hidden/read.run
>>>> > ====== BEGIN OUTPUT ======
>>>> > beast.http.read
>>>> > ThreadSanitizer:DEADLYSIGNAL
>>>> > ==1761540==ERROR: ThreadSanitizer: SEGV on unknown address
>>>> > 0x7f6ea78ff000 (pc 0x7f6eaaaba0d0 bp 0x000000000000 sp 0x7f6ea783d910
>>>> > T1761542)
>>>> > ==1761540==The signal is caused by a READ memory access.
>>>> >    #0 __sanitizer::StackDepotBase<__sanitizer::StackDepotNode, 1,
>>>> > 20>::Put(__sanitizer::StackTrace, bool*) <null> (libtsan.so.2+0xba0d0)
>>>> >    #1 __tsan::CurrentStackId(__tsan::ThreadState*, unsigned long)
>>>> > <null> (libtsan.so.2+0x8c48f)
>>>> >    #2 __sanitizer::DD::MutexInit(__sanitizer::DDCallback*,
>>>> > __sanitizer::DDMutex*) <null> (libtsan.so.2+0xac534)
>>>> >    #3 __tsan::DDMutexInit(__tsan::ThreadState*, unsigned long,
>>>> > __tsan::SyncVar*) <null> (libtsan.so.2+0x9a3f8)
>>>> >    #4 __tsan::MetaMap::GetSync(__tsan::ThreadState*, unsigned long,
>>>> > unsigned long, bool, bool) <null> (libtsan.so.2+0xa85dc)
>>>> >    #5 __tsan_atomic32_fetch_add <null> (libtsan.so.2+0x783e9)
>>>> >    #6 __gnu_cxx::__exchange_and_add(int volatile*, int)
>>>> > /usr/include/c++/12/ext/atomicity.h:66 (read+0x41ad46)
>>>> >    #7 __gnu_cxx::__exchange_and_add_dispatch(int*, int)
>>>> > /usr/include/c++/12/ext/atomicity.h:101 (read+0x41ad46)
>>>> >    #8 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release_last_use()
>>>> > /usr/include/c++/12/bits/shared_ptr_base.h:187 (read+0x41ad46)
>>>> >    #9 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release()
>>>> > /usr/include/c++/12/bits/shared_ptr_base.h:361 (read+0x40d87c)
>>>> >    #10 std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count()
>>>> > /usr/include/c++/12/bits/shared_ptr_base.h:1071 (read+0x41b43c)
>>>> >    #11 std::__shared_ptr<boost::asio::detail::strand_executor_service::strand_impl,
>>>> > (__gnu_cxx::_Lock_policy)2>::~__shared_ptr()
>>>> > /usr/include/c++/12/bits/shared_ptr_base.h:1524 (read+0x424db7)
>>>> >    #12 std::shared_ptr<boost::asio::detail::strand_executor_service::strand_impl>::~shared_ptr()
>>>> > /usr/include/c++/12/bits/shared_ptr.h:175 (read+0x424de3)
>>>> >    #13 boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>,
>>>> > 0ul> const, void>::~invoker() <null> (read+0x486f4d)
>>>> >    #14 boost::asio::detail::executor_op<boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>,
>>>> > 0ul> const, void>, boost::asio::detail::recycling_allocator<void,
>>>> > boost::asio::detail::thread_info_base::default_tag>,
>>>> > boost::asio::detail::scheduler_operation>::do_complete(void*,
>>>> > boost::asio::detail::scheduler_operation*, boost::system::error_code
>>>> > const&, unsigned long) <null> (read+0x49ff36)
>>>> >    #15 boost::asio::detail::scheduler_operation::complete(void*,
>>>> > boost::system::error_code const&, unsigned long)
>>>> > boost/asio/detail/scheduler_operation.hpp:40 (read+0x513742)
>>>> >    #16 boost::asio::detail::scheduler::do_run_one(boost::asio::detail::conditionally_enabled_mutex::scoped_lock&,
>>>> > boost::asio::detail::scheduler_thread_info&, boost::system::error_code
>>>> > const&) boost/asio/detail/impl/scheduler.ipp:492 (read+0x501be9)
>>>> >    #17 boost::asio::detail::scheduler::run(boost::system::error_code&)
>>>> > boost/asio/detail/impl/scheduler.ipp:210 (read+0x5008af)
>>>> >    #18 boost::asio::io_context::run()
>>>> > boost/asio/impl/io_context.ipp:63 (read+0x4f54d6)
>>>> >    #19 boost::beast::test::enable_yield_to::enable_yield_to(unsigned
>>>> > long)::{lambda()#1}::operator()() const <null> (read+0x41480d)
>>>> >    #20 void std::__invoke_impl<void,
>>>> > boost::beast::test::enable_yield_to::enable_yield_to(unsigned
>>>> > long)::{lambda()#1}>(std::__invoke_other,
>>>> > boost::beast::test::enable_yield_to::enable_yield_to(unsigned
>>>> > long)::{lambda()#1}&&) <null> (read+0x4bb4bc)
>>>> >    #21 std::__invoke_result<boost::beast::test::enable_yield_to::enable_yield_to(unsigned
>>>> > long)::{lambda()#1}>::type
>>>> > std::__invoke<boost::beast::test::enable_yield_to::enable_yield_to(unsigned
>>>> > long)::{lambda()#1}>(boost::beast::test::enable_yield_to::enable_yield_to(unsigned
>>>> > long)::{lambda()#1}&&) <null> (read+0x4b8c78)
>>>> >    #22 void std::thread::_Invoker<std::tuple<boost::beast::test::enable_yield_to::enable_yield_to(unsigned
>>>> > long)::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) <null>
>>>> > (read+0x4b5b00)
>>>> >    #23 std::thread::_Invoker<std::tuple<boost::beast::test::enable_yield_to::enable_yield_to(unsigned
>>>> > long)::{lambda()#1}> >::operator()() <null> (read+0x4b1f74)
>>>> >    #24 std::thread::_State_impl<std::thread::_Invoker<std::tuple<boost::beast::test::enable_yield_to::enable_yield_to(unsigned
>>>> > long)::{lambda()#1}> > >::_M_run() <null> (read+0x4ab02e)
>>>> >    #25 execute_native_thread_routine <null> (libstdc++.so.6+0xdbb72)
>>>> >    #26 __tsan_thread_start_func <null> (libtsan.so.2+0x393ef)
>>>> >    #27 start_thread <null> (libc.so.6+0x8ce2c)
>>>> >    #28 clone3 <null> (libc.so.6+0x1121af)
>>>> > 
>>>> > ThreadSanitizer can not provide additional info.
>>>> > SUMMARY: ThreadSanitizer: SEGV (/lib64/libtsan.so.2+0xba0d0) in
>>>> > __sanitizer::StackDepotBase<__sanitizer::StackDepotNode, 1,
>>>> > 20>::Put(__sanitizer::StackTrace, bool*)
>>>> > ==1761540==ABORTING
>>>> > 
>>>> > EXIT STATUS: 66
>>>> > ====== END OUTPUT ======
>>>> > 
>>>> > 
>>>> > 
>>>> > 
>>>> > -- 
>>>> > Richard Hodges
>>>> > hodges.r_at_[hidden]
>>>> > office: [phone]
>>>> > home: [phone]
>>>> > mobile: [phone]
>>>> > 
>>>> > _______________________________________________
>>>> > Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost