So sometime on Dec 27, my test runners for master stopped running. It appears to be fine for other runners, so I guess it is something in my setup, but after hunting around for a while I don't have any idea what....

The end result that I'm seeing is a failure of process_jam_log to build. 

Here's part of of the output from run.py/regression.py (started from my own python scripts):

# Searching for "b2" in "/var/boost/run/boost_bb/src/engine"...
# b2 succesfully built in "/var/boost/run/boost_bb/src/engine/bin.linuxx86_64/b2" location
# Preinstalled "/var/boost/run/boost_regression/stage/bin/process_jam_log" not found; building one...
# Found "process_jam_log" source directory "/var/boost/run/boost_regression/testing/build"
# Building "process_jam_log" ("/var/boost/run/boost_bb/src/engine/bin.linuxx86_64/b2" "-sBOOST_BUILD_PATH=/var/boost/run:/var/boost/run/boost_bb/src" "-sBOOST_ROOT=/var/boost/run/boost_root" "--boost=/var/boost/run/boost_root" "--boost-root=/var/boost/run/boost_root" "--boost-build=/var/boost/run/boost_bb/src" "--debug-configuration" install -d2 -l300)...
notice: found boost-build.jam at /var/boost/run/boost_regression/boost-build.jam
notice: loading Boost.Build from /var/boost/run/boost_bb/src
notice: Searching '/etc' '/home/boost' '/var/boost/run/boost_bb/src' '/var/boost/run' '/var/boost/run/boost_bb/src' '/var/boost/run/boost_bb/src/kernel' '/var/boost/run/boost_bb/src/util' '/var/boost/run/boost_bb/src/build' '/var/boost/run/boost_bb/src/tools' '/var/boost/run/boost_bb/src/contrib' '/var/boost/run/boost_bb/src/.' for site-config configuration file 'site-config.jam'.
notice: Configuration file 'site-config.jam' not found in '/etc' '/home/boost' '/var/boost/run/boost_bb/src' '/var/boost/run' '/var/boost/run/boost_bb/src' '/var/boost/run/boost_bb/src/kernel' '/var/boost/run/boost_bb/src/util' '/var/boost/run/boost_bb/src/build' '/var/boost/run/boost_bb/src/tools' '/var/boost/run/boost_bb/src/contrib' '/var/boost/run/boost_bb/src/.'.
notice: Searching '/home/boost' '/var/boost/run/boost_bb/src' '/var/boost/run' '/var/boost/run/boost_bb/src' '/var/boost/run/boost_bb/src/kernel' '/var/boost/run/boost_bb/src/util' '/var/boost/run/boost_bb/src/build' '/var/boost/run/boost_bb/src/tools' '/var/boost/run/boost_bb/src/contrib' '/var/boost/run/boost_bb/src/.' for user-config configuration file 'user-config.jam'.
notice: Loading user-config configuration file 'user-config.jam' from '/home/boost'.
notice: will use 'g++-7' for gcc, condition <toolset>gcc-7
notice: using gcc libraries :: <toolset>gcc-7 :: /usr/bin /usr/lib /usr/lib32 /usr/lib64
notice: using gcc archiver :: <toolset>gcc-7 :: /usr/bin/ar
notice: using gcc ranlib :: <toolset>gcc-7 :: /usr/bin/ranlib
warning: toolset gcc initialization: can not find tool windres
warning: initialized from /home/boost/user-config.jam:1
notice: using rc compiler :: <toolset>gcc-7 :: /usr/bin/as
notice: will use 'g++-7' for gcc, condition <toolset>gcc-7~c++98
notice: using gcc libraries :: <toolset>gcc-7~c++98 :: /usr/bin /usr/lib /usr/lib32 /usr/lib64
notice: using gcc archiver :: <toolset>gcc-7~c++98 :: /usr/bin/ar
notice: using gcc ranlib :: <toolset>gcc-7~c++98 :: /usr/bin/ranlib
warning: toolset gcc initialization: can not find tool windres
warning: initialized from /home/boost/user-config.jam:2
notice: using rc compiler :: <toolset>gcc-7~c++98 :: /usr/bin/as
notice: will use 'g++-7' for gcc, condition <toolset>gcc-7~gnu98
notice: using gcc libraries :: <toolset>gcc-7~gnu98 :: /usr/bin /usr/lib /usr/lib32 /usr/lib64
notice: using gcc archiver :: <toolset>gcc-7~gnu98 :: /usr/bin/ar
notice: using gcc ranlib :: <toolset>gcc-7~gnu98 :: /usr/bin/ranlib
warning: toolset gcc initialization: can not find tool windres
warning: initialized from /home/boost/user-config.jam:3
notice: using rc compiler :: <toolset>gcc-7~gnu98 :: /usr/bin/as
notice: will use 'g++-7' for gcc, condition <toolset>gcc-7~c++11
notice: using gcc libraries :: <toolset>gcc-7~c++11 :: /usr/bin /usr/lib /usr/lib32 /usr/lib64
notice: using gcc archiver :: <toolset>gcc-7~c++11 :: /usr/bin/ar
notice: using gcc ranlib :: <toolset>gcc-7~c++11 :: /usr/bin/ranlib
warning: toolset gcc initialization: can not find tool windres
warning: initialized from /home/boost/user-config.jam:4
notice: using rc compiler :: <toolset>gcc-7~c++11 :: /usr/bin/as
notice: will use 'g++-7' for gcc, condition <toolset>gcc-7~gnu11
notice: using gcc libraries :: <toolset>gcc-7~gnu11 :: /usr/bin /usr/lib /usr/lib32 /usr/lib64
notice: using gcc archiver :: <toolset>gcc-7~gnu11 :: /usr/bin/ar
notice: using gcc ranlib :: <toolset>gcc-7~gnu11 :: /usr/bin/ranlib
warning: toolset gcc initialization: can not find tool windres
warning: initialized from /home/boost/user-config.jam:5
notice: using rc compiler :: <toolset>gcc-7~gnu11 :: /usr/bin/as
notice: will use 'g++-7' for gcc, condition <toolset>gcc-7~c++14
notice: using gcc libraries :: <toolset>gcc-7~c++14 :: /usr/bin /usr/lib /usr/lib32 /usr/lib64
notice: using gcc archiver :: <toolset>gcc-7~c++14 :: /usr/bin/ar
notice: using gcc ranlib :: <toolset>gcc-7~c++14 :: /usr/bin/ranlib
warning: toolset gcc initialization: can not find tool windres
warning: initialized from /home/boost/user-config.jam:6
notice: using rc compiler :: <toolset>gcc-7~c++14 :: /usr/bin/as
notice: will use 'g++-7' for gcc, condition <toolset>gcc-7~gnu14
notice: using gcc libraries :: <toolset>gcc-7~gnu14 :: /usr/bin /usr/lib /usr/lib32 /usr/lib64
notice: using gcc archiver :: <toolset>gcc-7~gnu14 :: /usr/bin/ar
notice: using gcc ranlib :: <toolset>gcc-7~gnu14 :: /usr/bin/ranlib
warning: toolset gcc initialization: can not find tool windres
warning: initialized from /home/boost/user-config.jam:7
notice: using rc compiler :: <toolset>gcc-7~gnu14 :: /usr/bin/as
notice: will use 'g++-7' for gcc, condition <toolset>gcc-7~c++1z
notice: using gcc libraries :: <toolset>gcc-7~c++1z :: /usr/bin /usr/lib /usr/lib32 /usr/lib64
notice: using gcc archiver :: <toolset>gcc-7~c++1z :: /usr/bin/ar
notice: using gcc ranlib :: <toolset>gcc-7~c++1z :: /usr/bin/ranlib
warning: toolset gcc initialization: can not find tool windres
warning: initialized from /home/boost/user-config.jam:8
notice: using rc compiler :: <toolset>gcc-7~c++1z :: /usr/bin/as
notice: will use 'g++-7' for gcc, condition <toolset>gcc-7~gnu1z
notice: using gcc libraries :: <toolset>gcc-7~gnu1z :: /usr/bin /usr/lib /usr/lib32 /usr/lib64
notice: using gcc archiver :: <toolset>gcc-7~gnu1z :: /usr/bin/ar
notice: using gcc ranlib :: <toolset>gcc-7~gnu1z :: /usr/bin/ranlib
warning: toolset gcc initialization: can not find tool windres
warning: initialized from /home/boost/user-config.jam:9
notice: using rc compiler :: <toolset>gcc-7~gnu1z :: /usr/bin/as
notice: will use 'g++-7' for gcc, condition <toolset>gcc-7~c++1z~O2
notice: using gcc libraries :: <toolset>gcc-7~c++1z~O2 :: /usr/bin /usr/lib /usr/lib32 /usr/lib64
notice: using gcc archiver :: <toolset>gcc-7~c++1z~O2 :: /usr/bin/ar
notice: using gcc ranlib :: <toolset>gcc-7~c++1z~O2 :: /usr/bin/ranlib
warning: toolset gcc initialization: can not find tool windres
warning: initialized from /home/boost/user-config.jam:10
notice: using rc compiler :: <toolset>gcc-7~c++1z~O2 :: /usr/bin/as
notice: will use 'g++-7' for gcc, condition <toolset>gcc-7~c++1z~warn
notice: using gcc libraries :: <toolset>gcc-7~c++1z~warn :: /usr/bin /usr/lib /usr/lib32 /usr/lib64
notice: using gcc archiver :: <toolset>gcc-7~c++1z~warn :: /usr/bin/ar
notice: using gcc ranlib :: <toolset>gcc-7~c++1z~warn :: /usr/bin/ranlib
warning: toolset gcc initialization: can not find tool windres
warning: initialized from /home/boost/user-config.jam:11
notice: using rc compiler :: <toolset>gcc-7~c++1z~warn :: /usr/bin/as
notice: [python-cfg] Configuring python...
notice: [python-cfg]   user-specified version: "2.7"
notice: [python-cfg]   user-specified includes: "/usr/include/python2.7"
notice: [python-cfg]   user-specified libraries: "/usr/lib/x86_64-linux-gnu"
notice: [python-cfg] Checking interpreter command "python2.7"...
notice: [python-cfg] running command 'python2.7 -c "from sys import *; print('version=%d.%d\nplatform=%s\nprefix=%s\nexec_prefix=%s\nexecutable=%s' % (version_info[0],version_info[1],platform,prefix,exec_prefix,executable))" 2>&1'
notice: [python-cfg] ...requested configuration matched!
notice: [python-cfg] Details of this Python configuration:
notice: [python-cfg]   interpreter command: "python2.7"
notice: [python-cfg]   include path: "/usr/include/python2.7"
notice: [python-cfg]   library path: "/usr/lib/x86_64-linux-gnu"
notice: [python-cfg] Checking for NumPy...
notice: [python-cfg] running command 'python2.7 -c "import sys; sys.stderr = sys.stdout; import numpy; print(numpy.get_include())"'
notice: [python-cfg] NumPy disabled. Reason:
notice: [python-cfg]   python2.7 -c "import sys; sys.stderr = sys.stdout; import numpy; print(numpy.get_include())" aborted with 
notice: [python-cfg]   Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named numpy
notice: [python-cfg] Configuring python...
notice: [python-cfg]   user-specified version: "3.5"
notice: [python-cfg]   user-specified includes: "/usr/include/python3.5"
notice: [python-cfg]   user-specified libraries: "/usr/lib/x86_64-linux-gnu"
notice: [python-cfg] Checking interpreter command "python3.5"...
notice: [python-cfg] running command 'python3.5 -c "from sys import *; print('version=%d.%d\nplatform=%s\nprefix=%s\nexec_prefix=%s\nexecutable=%s' % (version_info[0],version_info[1],platform,prefix,exec_prefix,executable))" 2>&1'
notice: [python-cfg] ...requested configuration matched!
notice: [python-cfg] Details of this Python configuration:
notice: [python-cfg]   interpreter command: "python3.5"
notice: [python-cfg]   include path: "/usr/include/python3.5"
notice: [python-cfg]   library path: "/usr/lib/x86_64-linux-gnu"
notice: [python-cfg] Checking for NumPy...
notice: [python-cfg] running command 'python3.5 -c "import sys; sys.stderr = sys.stdout; import numpy; print(numpy.get_include())"'
notice: [python-cfg] NumPy disabled. Reason:
notice: [python-cfg]   python3.5 -c "import sys; sys.stderr = sys.stdout; import numpy; print(numpy.get_include())" aborted with 
notice: [python-cfg]   Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named 'numpy'
notice: [zlib] Using pre-installed library
notice: [zlib] Condition
notice: [bzip2] Using pre-installed library
notice: [bzip2] Condition
notice: [lzma] Using pre-installed library
notice: [lzma] Condition
Performing configuration checks

    - 32-bit                   : no
    - 64-bit                   : yes
    - arm                      : no
    - mips1                    : no
    - power                    : no
    - sparc                    : no
    - x86                      : yes
    - symlinks supported       : yes
error: Cannot create link ../../../boost_root/boost/functional/hash_fwd.hpp to ../../../boost_root/libs/container_hash/include/boost/functional/hash_fwd.hpp.
error: Link previously defined to another file, ../../../boost_root/libs/functional/include/boost/functional/hash_fwd.hpp.

# Searching for "process_jam_log" in "/var/boost/run/boost_regression/stage/bin"...
Traceback (most recent call last):
  File "run.py", line 71, in <module>
    runner(root)
  File "/var/boost/run/boost_regression_src/regression.py", line 286, in __init__
    self.main()
  File "/var/boost/run/boost_regression_src/regression.py", line 623, in main
    getattr(self,action_m)()
  File "/var/boost/run/boost_regression_src/regression.py", line 580, in command_regression
    self.command_setup()
  File "/var/boost/run/boost_regression_src/regression.py", line 351, in command_setup
    self.build_if_needed(self.process_jam_log,self.pjl_toolset)
  File "/var/boost/run/boost_regression_src/regression.py", line 714, in build_if_needed
    tool[ 'build_path' ] = self.tool_path( tool )
  File "/var/boost/run/boost_regression_src/regression.py", line 739, in tool_path
    , '\n'.join( [ name_or_spec[ 'path' ], build_dir ] )
Exception: Cannot find "process_jam_log" in any of the following locations:
/var/boost/run/boost_regression/stage/bin/process_jam_log
/var/boost/run/boost_regression/stage/bin

Everything looks fine until it goes to start making the symlinks (this test was run on linux, but I'm seeing the same issue on windows). For instance a good version of this on develop looks like:

# b2 succesfully built in "/var/boost/run/boost_bb/src/engine/bin.linuxx86_64/b2" location
# Preinstalled "/var/boost/run/boost_regression/stage/bin/process_jam_log" not found; building one...
# Found "process_jam_log" source directory "/var/boost/run/boost_regression/testing/build"
# Building "process_jam_log" ("/var/boost/run/boost_bb/src/engine/bin.linuxx86_64/b2" "-sBOOST_BUILD_PATH=/var/boost/run:/var/boost/run/boost_bb/src" "-sBOOST_ROOT=/var/boost/run/boost_root" "--boost=/var/boost/run/boost_root" "--boost-root=/var/boost/run/boost_root" "--boost-build=/var/boost/run/boost_bb/src" "--debug-configuration" install -d2 -l300)...
notice: found boost-build.jam at /var/boost/run/boost_regression/boost-build.jam
notice: loading Boost.Build from /var/boost/run/boost_bb/src
[snip]
notice: [lzma] Using pre-installed library
notice: [lzma] Condition
Performing configuration checks

    - 32-bit                   : no
    - 64-bit                   : yes
    - arm                      : no
    - mips1                    : no
    - power                    : no
    - sparc                    : no
    - x86                      : yes
    - symlinks supported       : yes
...found 1345 targets...
...updating 592 targets...
common.mkdir ../../bin

        mkdir -p "../../bin"
    
common.mkdir ../../bin/common

        mkdir -p "../../bin/common"
    
common.mkdir ../../bin/common/build

        mkdir -p "../../bin/common/build"
    
common.mkdir ../../bin/common/build/gcc-gnu-7

        mkdir -p "../../bin/common/build/gcc-gnu-7"
    
common.mkdir ../../bin/common/build/gcc-gnu-7/release

        mkdir -p "../../bin/common/build/gcc-gnu-7/release"
    
mklink-or-dir ../../../boost_root/boost

    mkdir "../../../boost_root/boost"

mklink-or-dir ../../../boost_root/boost/xpressive

    ln -s ../libs/xpressive/include/boost/xpressive ../../../boost_root/boost/xpressive

mklink-or-dir ../../../boost_root/boost/winapi

    ln -s ../libs/winapi/include/boost/winapi ../../../boost_root/boost/winapi
[snip]
common.copy /var/boost/run/boost_regression/stage/bin/process_jam_log

    cp "../../bin/testing/build/gcc-gnu-7/release/process_jam_log"  "/var/boost/run/boost_regression/stage/bin/process_jam_log"

...updated 592 targets...
# process_jam_log succesfully built in "/var/boost/run/boost_regression/stage/bin/process_jam_log" location
# Making "/var/boost/run/results" directory...



Any ideas why this error would be happening only on master? Different version of boost build (master vs. develop)?

Thanks,
Tom