Re: [Boost-testing] Test Failures on develop

From: Tom Kent (lists_at_[hidden])
Date: 2019-06-24 00:37:03

On Fri, Jun 21, 2019 at 5:11 AM Tom Kent <lists_at_[hidden]> wrote:

>
>
> On Thu, Jun 20, 2019, 10:35 PM Rene Rivera <grafikrobot_at_[hidden]> wrote:
>
>> On Thu, Jun 20, 2019 at 10:28 PM Tom Kent <lists_at_[hidden]> wrote:
>>
>>>
>>> On Thu, Jun 20, 2019 at 7:15 AM Rene Rivera <grafikrobot_at_[hidden]>
>>> wrote:
>>>
>>>> Since it's not due to the b2 -j change..
>>>>
>>>> On Fri, Jun 14, 2019 at 8:25 AM Tom Kent via Boost-Testing <
>>>> boost-testing_at_[hidden]> wrote:
>>>>
>>>>> For the last few days (Since June 4?) I've been having problems with
>>>>> all my linux tests running against the develop branch.
>>>>>
>>>>> GCC ends with this:
>>>>>
>>>>> gcc.compile.c++ ../../bin/common/build/gcc-8/release/process_jam_log.o
>>>>>
>>>>>    "g++-8"   -fPIC -O3 -finline-functions -Wno-inline -Wall
>>>>>  -DBOOST_ALL_NO_LIB=1 -DNDEBUG -D_CRT_SECURE_NO_WARNINGS  -
>>>>> I"/var/boost/run/boost_root" -c -o
>>>>> "../../bin/common/build/gcc-8/release/process_jam_log.o"
>>>>> "../../common/build/../src/p
>>>>> rocess_jam_log.cpp"
>>>>>
>>>>> ../../common/build/../src/process_jam_log.cpp: In function
>>>>> 'std::__cxx11::string {anonymous}::revision(const string&)':
>>>>> ../../common/build/../src/process_jam_log.cpp:359:16: warning:
>>>>> ignoring return value of 'int system(const char*)', decla
>>>>> red with attribute warn_unused_result [-Wunused-result]
>>>>>     std::system("git rev-parse --short=6 HEAD >.short-sha");
>>>>>     ~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>>> ...skipped <p../../bin/testing/build/gcc-8/release>process_jam_log for
>>>>> lack of <p/var/boost/run/boost_root/bin.v2/libs/f
>>>>> ilesystem/build/gcc-8/release/link-static/visibility-hidden>libboost_filesystem.a...
>>>>>
>>>>>
>>>>
>>>> I guess you'd need to find out what it says for trying to build
>>>> filesystem lib.
>>>>
>>>
>>>  Here's what I'm seeing with the filesystem error:
>>>
>>>    ln -f -s
>>> "../../libs/detail/include/boost/detail/has_default_constructor.hpp"
>>> "../../../boost_root/boost/detail/has_
>>> default_constructor.hpp"
>>>
>>> gcc.compile.c++
>>> /var/boost/run/boost_root/bin.v2/libs/filesystem/build/gcc-8/release/link-static/visibility-hidden/path_traits.o
>>>
>>>    "g++-8"   -fvisibility-inlines-hidden -m64 -O3 -finline-functions
>>> -Wno-inline -Wall -fvisibility=hidden  -DBOOST_ALL
>>> _NO_LIB=1 -DBOOST_FILESYSTEM_STATIC_LINK=1 -DNDEBUG
>>>  -I"/var/boost/run/boost_root" -c -o "/var/boost/run/boost_root/bin.
>>> v2/libs/filesystem/build/gcc-8/release/link-static/visibility-hidden/path_traits.o"
>>> "/var/boost/run/boost_root/libs/file
>>> system/src/path_traits.cpp"
>>>
>>> /var/boost/run/boost_root/libs/filesystem/src/path_traits.cpp:20:10:
>>> fatal error: boost/filesystem/config.hpp: No such f
>>> ile or directory
>>> #include <boost/filesystem/config.hpp>
>>>          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>> compilation terminated.
>>> ...failed gcc.compile.c++
>>> /var/boost/run/boost_root/bin.v2/libs/filesystem/build/gcc-8/release/link-static/visibility-hidden/path_traits.o...
>>> gcc.compile.c++
>>> /var/boost/run/boost_root/bin.v2/libs/filesystem/build/gcc-8/release/link-static/visibility-hidden/portability.o
>>>
>>>    "g++-8"   -fvisibility-inlines-hidden -m64 -O3 -finline-functions
>>> -Wno-inline -Wall -fvisibility=hidden  -DBOOST_ALL
>>> _NO_LIB=1 -DBOOST_FILESYSTEM_STATIC_LINK=1 -DNDEBUG
>>>  -I"/var/boost/run/boost_root" -c -o "/var/boost/run/boost_root/bin.
>>> v2/libs/filesystem/build/gcc-8/release/link-static/visibility-hidden/portability.o"
>>> "/var/boost/run/boost_root/libs/file
>>> system/src/portability.cpp"
>>>
>>> /var/boost/run/boost_root/libs/filesystem/src/portability.cpp:20:10:
>>> fatal error: boost/filesystem/config.hpp: No such f
>>> ile or directory
>>> #include <boost/filesystem/config.hpp>
>>>          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>> compilation terminated.
>>> ...failed gcc.compile.c++
>>> /var/boost/run/boost_root/bin.v2/libs/filesystem/build/gcc-8/release/link-static/visibility-hidden/portability.o...
>>>
>>>
>>> And several more like that.
>>>
>>> However, I'm also suspicious of the timing with the processor change.
>>>
>>> Looking back at the last successful builds of develop on linux. I have a
>>> few from linux/x86_64/docker machines. The last one I saw was building with
>>> gcc-8 on June 4 around 6:30am CDT. There should have been a successful
>>> clang run about 5hrs after that (11:30am CDT), and another successful gcc
>>> around 10hrs after the last one that was actually successful (4:30pm CDT).
>>>
>>> This seems like it would correspond to the change to build here:
>>>
>>> https://github.com/boostorg/build/commit/451059949d92b52b21f802562a50587f1eeecb82
>>>
>>> The other failure I started seeing was only my linux/arm64 machine. This
>>> doesn't run docker, so might not be affected in the same way. The last
>>> successful run on there was at 4:50pm CDT on June 4...but it typically
>>> takes around 9hrs to run so it probably got the pre 451059949 commit.
>>> However here I can't rule out that it was impacted by a8ab76ef973.
>>>
>>> I'm going to try to standup some normal VMs soon and run on them to see
>>> if they are affected in the same way. It would also be nice if we could try
>>> backing out the usage of _SC_NPROCESSORS_ONLN and see if that fixes things,
>>> is that feasible to try for a couple days?
>>>
>>
>> Again.. It's not a b2 issue. Specifying the "-jX" option entirely
>> obviates the changed code. It's almost certainly something else. Places to
>> look would be in the filesystem lib, or the root "boost" project for
>> changes.
>>
>
> Looking at regression.py it looks like the command that builds
> process_jam_log doesn't use the bjam options. Can you verify if this
> is/isn't the case?
>
> Tom
>

Somewhere between  9:57am and 12:02pm CDT today this bug migrated to master.

It can trivially be re-produced.

On a bare linux vm...Azure F8s_v2, Ubuntu 18.04, standard SSD.
$ sudo apt update && sudo apt install build-essential gcc-8 g++-8 python
python3 git wget
...
$ wget
https://raw.githubusercontent.com/boostorg/regression/develop/testing/src/run.py
$ echo using gcc : 8 : g++-8 : \; > ~/user-config.jam
$ python run.py --runner=teeks99-test01 --toolsets=gcc-8 --tag=develop
--bjam-options="-j8 address-model=64"
...fails with the above errors.

You can also do the same thing to reproduce in a docker image
(ubuntu:bionic). --bjam-options of -j1 -j8 and absent all fail the same.

This does not correspond to any changes between the filesystem library that
indicates the error. But it *does* correspond exactly with the release of
the new B2.
https://github.com/boostorg/build/commit/04efaac148671479b514db882b1c520dcaafe89e

There have been no successful regression tests by any linux runners since
that time.

Tom