TSAN SEGFAULT in Boost.Beast CI

From: Richard Hodges (hodges.r_at_[hidden])
Date: 2022-08-12 18:24:48

Dear all,

You may have noticed that one of the Beast CI checks fails on the master
and develop branches.

The failure is a segfault in thread sanitizer. Running on my local machine
I get the stack trace embedded below.

This does not happen if Asio is reverted back to version 1.79.0

Chris has confirmed that the change in Asio that caused this is in
asio::spawn.

Note that at this stage there is no reason to suspect that Asio is actually
at fault. I believe it to be a false positive. However, I am an absolute
noob when it comes to the Dark Arts of thread sanitizer. I have no idea
what to blacklist.

This email has two aims:

1. To prevent multiple people wasting their time trying to get to the
bottom of this,
2. To encourage some sage TSAN-maester to emerge who may guide me to
enlightenment.

Here is the error and stack trace:

testing.capture-output
bin.v2/libs/beast/test/beast/http/read.test/gcc-12/debug/link-static/thread-sanitizer-norecover/threading-multi/visibility-hidden/read.run
====== BEGIN OUTPUT ======
beast.http.read
ThreadSanitizer:DEADLYSIGNAL
==1761540==ERROR: ThreadSanitizer: SEGV on unknown address
0x7f6ea78ff000 (pc 0x7f6eaaaba0d0 bp 0x000000000000 sp 0x7f6ea783d910
T1761542)
==1761540==The signal is caused by a READ memory access.
    #0 __sanitizer::StackDepotBase<__sanitizer::StackDepotNode, 1,
20>::Put(__sanitizer::StackTrace, bool*) <null> (libtsan.so.2+0xba0d0)
    #1 __tsan::CurrentStackId(__tsan::ThreadState*, unsigned long)
<null> (libtsan.so.2+0x8c48f)
    #2 __sanitizer::DD::MutexInit(__sanitizer::DDCallback*,
__sanitizer::DDMutex*) <null> (libtsan.so.2+0xac534)
    #3 __tsan::DDMutexInit(__tsan::ThreadState*, unsigned long,
__tsan::SyncVar*) <null> (libtsan.so.2+0x9a3f8)
    #4 __tsan::MetaMap::GetSync(__tsan::ThreadState*, unsigned long,
unsigned long, bool, bool) <null> (libtsan.so.2+0xa85dc)
    #5 __tsan_atomic32_fetch_add <null> (libtsan.so.2+0x783e9)
    #6 __gnu_cxx::__exchange_and_add(int volatile*, int)
/usr/include/c++/12/ext/atomicity.h:66 (read+0x41ad46)
    #7 __gnu_cxx::__exchange_and_add_dispatch(int*, int)
/usr/include/c++/12/ext/atomicity.h:101 (read+0x41ad46)
    #8 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release_last_use()
/usr/include/c++/12/bits/shared_ptr_base.h:187 (read+0x41ad46)
    #9 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release()
/usr/include/c++/12/bits/shared_ptr_base.h:361 (read+0x40d87c)
    #10 std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count()
/usr/include/c++/12/bits/shared_ptr_base.h:1071 (read+0x41b43c)
    #11 std::__shared_ptr<boost::asio::detail::strand_executor_service::strand_impl,
(__gnu_cxx::_Lock_policy)2>::~__shared_ptr()
/usr/include/c++/12/bits/shared_ptr_base.h:1524 (read+0x424db7)
    #12 std::shared_ptr<boost::asio::detail::strand_executor_service::strand_impl>::~shared_ptr()
/usr/include/c++/12/bits/shared_ptr.h:175 (read+0x424de3)
    #13 boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>,
0ul> const, void>::~invoker() <null> (read+0x486f4d)
    #14 boost::asio::detail::executor_op<boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>,
0ul> const, void>, boost::asio::detail::recycling_allocator<void,
boost::asio::detail::thread_info_base::default_tag>,
boost::asio::detail::scheduler_operation>::do_complete(void*,
boost::asio::detail::scheduler_operation*, boost::system::error_code
const&, unsigned long) <null> (read+0x49ff36)
    #15 boost::asio::detail::scheduler_operation::complete(void*,
boost::system::error_code const&, unsigned long)
boost/asio/detail/scheduler_operation.hpp:40 (read+0x513742)
    #16 boost::asio::detail::scheduler::do_run_one(boost::asio::detail::conditionally_enabled_mutex::scoped_lock&,
boost::asio::detail::scheduler_thread_info&, boost::system::error_code
const&) boost/asio/detail/impl/scheduler.ipp:492 (read+0x501be9)
    #17 boost::asio::detail::scheduler::run(boost::system::error_code&)
boost/asio/detail/impl/scheduler.ipp:210 (read+0x5008af)
    #18 boost::asio::io_context::run()
boost/asio/impl/io_context.ipp:63 (read+0x4f54d6)
    #19 boost::beast::test::enable_yield_to::enable_yield_to(unsigned
long)::{lambda()#1}::operator()() const <null> (read+0x41480d)
    #20 void std::__invoke_impl<void,
boost::beast::test::enable_yield_to::enable_yield_to(unsigned
long)::{lambda()#1}>(std::__invoke_other,
boost::beast::test::enable_yield_to::enable_yield_to(unsigned
long)::{lambda()#1}&&) <null> (read+0x4bb4bc)
    #21 std::__invoke_result<boost::beast::test::enable_yield_to::enable_yield_to(unsigned
long)::{lambda()#1}>::type
std::__invoke<boost::beast::test::enable_yield_to::enable_yield_to(unsigned
long)::{lambda()#1}>(boost::beast::test::enable_yield_to::enable_yield_to(unsigned
long)::{lambda()#1}&&) <null> (read+0x4b8c78)
    #22 void std::thread::_Invoker<std::tuple<boost::beast::test::enable_yield_to::enable_yield_to(unsigned
long)::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) <null>
(read+0x4b5b00)
    #23 std::thread::_Invoker<std::tuple<boost::beast::test::enable_yield_to::enable_yield_to(unsigned
long)::{lambda()#1}> >::operator()() <null> (read+0x4b1f74)
    #24 std::thread::_State_impl<std::thread::_Invoker<std::tuple<boost::beast::test::enable_yield_to::enable_yield_to(unsigned
long)::{lambda()#1}> > >::_M_run() <null> (read+0x4ab02e)
    #25 execute_native_thread_routine <null> (libstdc++.so.6+0xdbb72)
    #26 __tsan_thread_start_func <null> (libtsan.so.2+0x393ef)
    #27 start_thread <null> (libc.so.6+0x8ce2c)
    #28 clone3 <null> (libc.so.6+0x1121af)

ThreadSanitizer can not provide additional info.
SUMMARY: ThreadSanitizer: SEGV (/lib64/libtsan.so.2+0xba0d0) in
__sanitizer::StackDepotBase<__sanitizer::StackDepotNode, 1,
20>::Put(__sanitizer::StackTrace, bool*)
==1761540==ABORTING

EXIT STATUS: 66
====== END OUTPUT ======

-- 
Richard Hodges
hodges.r_at_[hidden]
office: [phone]
home: [phone]
mobile: [phone]