[Boost-build] My master regression runners failing

Subject: [Boost-build] My master regression runners failing
From: Tom Kent (lists_at_[hidden])
Date: 2018-01-01 15:37:40

So sometime on Dec 27, my test runners for master stopped running. It
appears to be fine for other runners, so I guess it is something in my
setup, but after hunting around for a while I don't have any idea what....

The end result that I'm seeing is a failure of process_jam_log to build.

Here's part of of the output from run.py/regression.py (started from my own
python scripts):

# Searching for "b2" in "/var/boost/run/boost_bb/src/engine"...
# b2 succesfully built in
"/var/boost/run/boost_bb/src/engine/bin.linuxx86_64/b2"
location
# Preinstalled "/var/boost/run/boost_regression/stage/bin/process_jam_log"
not found; building one...
# Found "process_jam_log" source directory "/var/boost/run/boost_
regression/testing/build"
# Building "process_jam_log"
("/var/boost/run/boost_bb/src/engine/bin.linuxx86_64/b2"
"-sBOOST_BUILD_PATH=/var/boost/run:/var/boost/run/boost_bb/src"
"-sBOOST_ROOT=/var/boost/run/boost_root" "--boost=/var/boost/run/boost_root"
"--boost-root=/var/boost/run/boost_root"
"--boost-build=/var/boost/run/boost_bb/src"
"--debug-configuration" install -d2 -l300)...
notice: found boost-build.jam at /var/boost/run/boost_
regression/boost-build.jam
notice: loading Boost.Build from /var/boost/run/boost_bb/src
notice: Searching '/etc' '/home/boost' '/var/boost/run/boost_bb/src'
'/var/boost/run' '/var/boost/run/boost_bb/src'
'/var/boost/run/boost_bb/src/kernel'
'/var/boost/run/boost_bb/src/util' '/var/boost/run/boost_bb/src/build'
'/var/boost/run/boost_bb/src/tools' '/var/boost/run/boost_bb/src/contrib'
'/var/boost/run/boost_bb/src/.' for site-config configuration file
'site-config.jam'.
notice: Configuration file 'site-config.jam' not found in '/etc'
'/home/boost' '/var/boost/run/boost_bb/src' '/var/boost/run'
'/var/boost/run/boost_bb/src' '/var/boost/run/boost_bb/src/kernel'
'/var/boost/run/boost_bb/src/util' '/var/boost/run/boost_bb/src/build'
'/var/boost/run/boost_bb/src/tools' '/var/boost/run/boost_bb/src/contrib'
'/var/boost/run/boost_bb/src/.'.
notice: Searching '/home/boost' '/var/boost/run/boost_bb/src'
'/var/boost/run' '/var/boost/run/boost_bb/src'
'/var/boost/run/boost_bb/src/kernel'
'/var/boost/run/boost_bb/src/util' '/var/boost/run/boost_bb/src/build'
'/var/boost/run/boost_bb/src/tools' '/var/boost/run/boost_bb/src/contrib'
'/var/boost/run/boost_bb/src/.' for user-config configuration file
'user-config.jam'.
notice: Loading user-config configuration file 'user-config.jam' from
'/home/boost'.
notice: will use 'g++-7' for gcc, condition <toolset>gcc-7
notice: using gcc libraries :: <toolset>gcc-7 :: /usr/bin /usr/lib
/usr/lib32 /usr/lib64
notice: using gcc archiver :: <toolset>gcc-7 :: /usr/bin/ar
notice: using gcc ranlib :: <toolset>gcc-7 :: /usr/bin/ranlib
warning: toolset gcc initialization: can not find tool windres
warning: initialized from /home/boost/user-config.jam:1
notice: using rc compiler :: <toolset>gcc-7 :: /usr/bin/as
notice: will use 'g++-7' for gcc, condition <toolset>gcc-7~c++98
notice: using gcc libraries :: <toolset>gcc-7~c++98 :: /usr/bin /usr/lib
/usr/lib32 /usr/lib64
notice: using gcc archiver :: <toolset>gcc-7~c++98 :: /usr/bin/ar
notice: using gcc ranlib :: <toolset>gcc-7~c++98 :: /usr/bin/ranlib
warning: toolset gcc initialization: can not find tool windres
warning: initialized from /home/boost/user-config.jam:2
notice: using rc compiler :: <toolset>gcc-7~c++98 :: /usr/bin/as
notice: will use 'g++-7' for gcc, condition <toolset>gcc-7~gnu98
notice: using gcc libraries :: <toolset>gcc-7~gnu98 :: /usr/bin /usr/lib
/usr/lib32 /usr/lib64
notice: using gcc archiver :: <toolset>gcc-7~gnu98 :: /usr/bin/ar
notice: using gcc ranlib :: <toolset>gcc-7~gnu98 :: /usr/bin/ranlib
warning: toolset gcc initialization: can not find tool windres
warning: initialized from /home/boost/user-config.jam:3
notice: using rc compiler :: <toolset>gcc-7~gnu98 :: /usr/bin/as
notice: will use 'g++-7' for gcc, condition <toolset>gcc-7~c++11
notice: using gcc libraries :: <toolset>gcc-7~c++11 :: /usr/bin /usr/lib
/usr/lib32 /usr/lib64
notice: using gcc archiver :: <toolset>gcc-7~c++11 :: /usr/bin/ar
notice: using gcc ranlib :: <toolset>gcc-7~c++11 :: /usr/bin/ranlib
warning: toolset gcc initialization: can not find tool windres
warning: initialized from /home/boost/user-config.jam:4
notice: using rc compiler :: <toolset>gcc-7~c++11 :: /usr/bin/as
notice: will use 'g++-7' for gcc, condition <toolset>gcc-7~gnu11
notice: using gcc libraries :: <toolset>gcc-7~gnu11 :: /usr/bin /usr/lib
/usr/lib32 /usr/lib64
notice: using gcc archiver :: <toolset>gcc-7~gnu11 :: /usr/bin/ar
notice: using gcc ranlib :: <toolset>gcc-7~gnu11 :: /usr/bin/ranlib
warning: toolset gcc initialization: can not find tool windres
warning: initialized from /home/boost/user-config.jam:5
notice: using rc compiler :: <toolset>gcc-7~gnu11 :: /usr/bin/as
notice: will use 'g++-7' for gcc, condition <toolset>gcc-7~c++14
notice: using gcc libraries :: <toolset>gcc-7~c++14 :: /usr/bin /usr/lib
/usr/lib32 /usr/lib64
notice: using gcc archiver :: <toolset>gcc-7~c++14 :: /usr/bin/ar
notice: using gcc ranlib :: <toolset>gcc-7~c++14 :: /usr/bin/ranlib
warning: toolset gcc initialization: can not find tool windres
warning: initialized from /home/boost/user-config.jam:6
notice: using rc compiler :: <toolset>gcc-7~c++14 :: /usr/bin/as
notice: will use 'g++-7' for gcc, condition <toolset>gcc-7~gnu14
notice: using gcc libraries :: <toolset>gcc-7~gnu14 :: /usr/bin /usr/lib
/usr/lib32 /usr/lib64
notice: using gcc archiver :: <toolset>gcc-7~gnu14 :: /usr/bin/ar
notice: using gcc ranlib :: <toolset>gcc-7~gnu14 :: /usr/bin/ranlib
warning: toolset gcc initialization: can not find tool windres
warning: initialized from /home/boost/user-config.jam:7
notice: using rc compiler :: <toolset>gcc-7~gnu14 :: /usr/bin/as
notice: will use 'g++-7' for gcc, condition <toolset>gcc-7~c++1z
notice: using gcc libraries :: <toolset>gcc-7~c++1z :: /usr/bin /usr/lib
/usr/lib32 /usr/lib64
notice: using gcc archiver :: <toolset>gcc-7~c++1z :: /usr/bin/ar
notice: using gcc ranlib :: <toolset>gcc-7~c++1z :: /usr/bin/ranlib
warning: toolset gcc initialization: can not find tool windres
warning: initialized from /home/boost/user-config.jam:8
notice: using rc compiler :: <toolset>gcc-7~c++1z :: /usr/bin/as
notice: will use 'g++-7' for gcc, condition <toolset>gcc-7~gnu1z
notice: using gcc libraries :: <toolset>gcc-7~gnu1z :: /usr/bin /usr/lib
/usr/lib32 /usr/lib64
notice: using gcc archiver :: <toolset>gcc-7~gnu1z :: /usr/bin/ar
notice: using gcc ranlib :: <toolset>gcc-7~gnu1z :: /usr/bin/ranlib
warning: toolset gcc initialization: can not find tool windres
warning: initialized from /home/boost/user-config.jam:9
notice: using rc compiler :: <toolset>gcc-7~gnu1z :: /usr/bin/as
notice: will use 'g++-7' for gcc, condition <toolset>gcc-7~c++1z~O2
notice: using gcc libraries :: <toolset>gcc-7~c++1z~O2 :: /usr/bin /usr/lib
/usr/lib32 /usr/lib64
notice: using gcc archiver :: <toolset>gcc-7~c++1z~O2 :: /usr/bin/ar
notice: using gcc ranlib :: <toolset>gcc-7~c++1z~O2 :: /usr/bin/ranlib
warning: toolset gcc initialization: can not find tool windres
warning: initialized from /home/boost/user-config.jam:10
notice: using rc compiler :: <toolset>gcc-7~c++1z~O2 :: /usr/bin/as
notice: will use 'g++-7' for gcc, condition <toolset>gcc-7~c++1z~warn
notice: using gcc libraries :: <toolset>gcc-7~c++1z~warn :: /usr/bin
/usr/lib /usr/lib32 /usr/lib64
notice: using gcc archiver :: <toolset>gcc-7~c++1z~warn :: /usr/bin/ar
notice: using gcc ranlib :: <toolset>gcc-7~c++1z~warn :: /usr/bin/ranlib
warning: toolset gcc initialization: can not find tool windres
warning: initialized from /home/boost/user-config.jam:11
notice: using rc compiler :: <toolset>gcc-7~c++1z~warn :: /usr/bin/as
notice: [python-cfg] Configuring python...
notice: [python-cfg]   user-specified version: "2.7"
notice: [python-cfg]   user-specified includes: "/usr/include/python2.7"
notice: [python-cfg]   user-specified libraries: "/usr/lib/x86_64-linux-gnu"
notice: [python-cfg] Checking interpreter command "python2.7"...
notice: [python-cfg] running command 'python2.7 -c "from sys import *;
print('version=%d.%d\nplatform=%s\nprefix=%s\nexec_prefix=%s\nexecutable=%s'
% (version_info[0],version_info[1],platform,prefix,exec_prefix,executable))"
2>&1'
notice: [python-cfg] ...requested configuration matched!
notice: [python-cfg] Details of this Python configuration:
notice: [python-cfg]   interpreter command: "python2.7"
notice: [python-cfg]   include path: "/usr/include/python2.7"
notice: [python-cfg]   library path: "/usr/lib/x86_64-linux-gnu"
notice: [python-cfg] Checking for NumPy...
notice: [python-cfg] running command 'python2.7 -c "import sys; sys.stderr
= sys.stdout; import numpy; print(numpy.get_include())"'
notice: [python-cfg] NumPy disabled. Reason:
notice: [python-cfg]   python2.7 -c "import sys; sys.stderr = sys.stdout;
import numpy; print(numpy.get_include())" aborted with
notice: [python-cfg]   Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named numpy
notice: [python-cfg] Configuring python...
notice: [python-cfg]   user-specified version: "3.5"
notice: [python-cfg]   user-specified includes: "/usr/include/python3.5"
notice: [python-cfg]   user-specified libraries: "/usr/lib/x86_64-linux-gnu"
notice: [python-cfg] Checking interpreter command "python3.5"...
notice: [python-cfg] running command 'python3.5 -c "from sys import *;
print('version=%d.%d\nplatform=%s\nprefix=%s\nexec_prefix=%s\nexecutable=%s'
% (version_info[0],version_info[1],platform,prefix,exec_prefix,executable))"
2>&1'
notice: [python-cfg] ...requested configuration matched!
notice: [python-cfg] Details of this Python configuration:
notice: [python-cfg]   interpreter command: "python3.5"
notice: [python-cfg]   include path: "/usr/include/python3.5"
notice: [python-cfg]   library path: "/usr/lib/x86_64-linux-gnu"
notice: [python-cfg] Checking for NumPy...
notice: [python-cfg] running command 'python3.5 -c "import sys; sys.stderr
= sys.stdout; import numpy; print(numpy.get_include())"'
notice: [python-cfg] NumPy disabled. Reason:
notice: [python-cfg]   python3.5 -c "import sys; sys.stderr = sys.stdout;
import numpy; print(numpy.get_include())" aborted with
notice: [python-cfg]   Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named 'numpy'
notice: [zlib] Using pre-installed library
notice: [zlib] Condition
notice: [bzip2] Using pre-installed library
notice: [bzip2] Condition
notice: [lzma] Using pre-installed library
notice: [lzma] Condition
Performing configuration checks

    - 32-bit                   : no
    - 64-bit                   : yes
    - arm                      : no
    - mips1                    : no
    - power                    : no
    - sparc                    : no
    - x86                      : yes
    - symlinks supported       : yes
error: Cannot create link ../../../boost_root/boost/functional/hash_fwd.hpp
to ../../../boost_root/libs/container_hash/include/boost/
functional/hash_fwd.hpp.
error: Link previously defined to another file, ../../../boost_root/libs/
functional/include/boost/functional/hash_fwd.hpp.

# Searching for "process_jam_log" in "/var/boost/run/boost_
regression/stage/bin"...
Traceback (most recent call last):
  File "run.py", line 71, in <module>
    runner(root)
  File "/var/boost/run/boost_regression_src/regression.py", line 286, in
__init__
    self.main()
  File "/var/boost/run/boost_regression_src/regression.py", line 623, in
main
    getattr(self,action_m)()
  File "/var/boost/run/boost_regression_src/regression.py", line 580, in
command_regression
    self.command_setup()
  File "/var/boost/run/boost_regression_src/regression.py", line 351, in
command_setup
    self.build_if_needed(self.process_jam_log,self.pjl_toolset)
  File "/var/boost/run/boost_regression_src/regression.py", line 714, in
build_if_needed
    tool[ 'build_path' ] = self.tool_path( tool )
  File "/var/boost/run/boost_regression_src/regression.py", line 739, in
tool_path
    , '\n'.join( [ name_or_spec[ 'path' ], build_dir ] )
Exception: Cannot find "process_jam_log" in any of the following locations:
/var/boost/run/boost_regression/stage/bin/process_jam_log
/var/boost/run/boost_regression/stage/bin

Everything looks fine until it goes to start making the symlinks (this test
was run on linux, but I'm seeing the same issue on windows). For instance a
good version of this on develop looks like:

# b2 succesfully built in
"/var/boost/run/boost_bb/src/engine/bin.linuxx86_64/b2"
location
# Preinstalled "/var/boost/run/boost_regression/stage/bin/process_jam_log"
not found; building one...
# Found "process_jam_log" source directory "/var/boost/run/boost_
regression/testing/build"
# Building "process_jam_log"
("/var/boost/run/boost_bb/src/engine/bin.linuxx86_64/b2"
"-sBOOST_BUILD_PATH=/var/boost/run:/var/boost/run/boost_bb/src"
"-sBOOST_ROOT=/var/boost/run/boost_root" "--boost=/var/boost/run/boost_root"
"--boost-root=/var/boost/run/boost_root"
"--boost-build=/var/boost/run/boost_bb/src"
"--debug-configuration" install -d2 -l300)...
notice: found boost-build.jam at /var/boost/run/boost_
regression/boost-build.jam
notice: loading Boost.Build from /var/boost/run/boost_bb/src
[snip]
notice: [lzma] Using pre-installed library
notice: [lzma] Condition
Performing configuration checks

    - 32-bit                   : no
    - 64-bit                   : yes
    - arm                      : no
    - mips1                    : no
    - power                    : no
    - sparc                    : no
    - x86                      : yes
    - symlinks supported       : yes
...found 1345 targets...
...updating 592 targets...
common.mkdir ../../bin

        mkdir -p "../../bin"

common.mkdir ../../bin/common

        mkdir -p "../../bin/common"

common.mkdir ../../bin/common/build

        mkdir -p "../../bin/common/build"

common.mkdir ../../bin/common/build/gcc-gnu-7

        mkdir -p "../../bin/common/build/gcc-gnu-7"

common.mkdir ../../bin/common/build/gcc-gnu-7/release

        mkdir -p "../../bin/common/build/gcc-gnu-7/release"

mklink-or-dir ../../../boost_root/boost

    mkdir "../../../boost_root/boost"

mklink-or-dir ../../../boost_root/boost/xpressive

    ln -s ../libs/xpressive/include/boost/xpressive
../../../boost_root/boost/xpressive

mklink-or-dir ../../../boost_root/boost/winapi

    ln -s ../libs/winapi/include/boost/winapi ../../../boost_root/boost/
winapi
[snip]
common.copy /var/boost/run/boost_regression/stage/bin/process_jam_log

    cp "../../bin/testing/build/gcc-gnu-7/release/process_jam_log"
"/var/boost/run/boost_regression/stage/bin/process_jam_log"

...updated 592 targets...
# process_jam_log succesfully built in "/var/boost/run/boost_
regression/stage/bin/process_jam_log" location
# Making "/var/boost/run/results" directory...

Any ideas why this error would be happening only on master? Different
version of boost build (master vs. develop)?

Thanks,
Tom