[Boost-build] Trying to build boost 1.71.0 on CentOS with Intel Compiler and custom gcc: has_icu test uses wrong library runtime path (for libstdc++)

From: Dr. Thomas Orgis (thomas.orgis_at_[hidden])
Date: 2019-09-02 17:14:47

Hi,

I manage a software setup with many separate prefixes and software
trees, selectable by the users via environment modules. The base system
is CentOS 7.6, but we try to only use very basic system software from
the distro. I am trying to build a scientific software that pretty much
requires the Intel compiler suite and Boost. We have the Intel
compilers version 19.0.4 installed. I am combining it with gcc 8.3.0 to
have a more modern libstdc++ available (the one from gcc 4.8 in CentOS
is a bit dated for Boost to be happy, or at least needs additional work
to enable C++11 support). 

My problem is: While I can successfully build, e.g. the regex library
with my desired compiler combination, the detection of ICU fails, as
the test binary misses the path to the correct libstdc++.

First, this is how it should look regarding libstdc++, and indeed does
for the boost lib:

[sw_at_node002 boost_1_71_0]$ ldd stage/lib/libboost_regex.so
        linux-vdso.so.1 =>  (0x00007ffdd00a2000)
        librt.so.1 => /lib64/librt.so.1 (0x0000145cb015d000)
        libimf.so => /sw/compiler/intel-19.0.4/compilers_and_libraries_2019.4.243/linux/compiler/lib/intel64_lin/libimf.so (0x0000145cafbbd000)
        libsvml.so => /sw/compiler/intel-19.0.4/compilers_and_libraries_2019.4.243/linux/compiler/lib/intel64_lin/libsvml.so (0x0000145cae219000)
        libirng.so => /sw/compiler/intel-19.0.4/compilers_and_libraries_2019.4.243/linux/compiler/lib/intel64_lin/libirng.so (0x0000145cadea7000)
        libstdc++.so.6 => /sw/compiler/gcc-8.3.0/lib64/libstdc++.so.6 (0x0000145cadd1b000)
        libm.so.6 => /lib64/libm.so.6 (0x0000145cada19000)
        libgcc_s.so.1 => /sw/compiler/gcc-8.3.0/lib64/libgcc_s.so.1 (0x0000145cad9ff000)
        libintlc.so.5 => /sw/compiler/intel-19.0.4/compilers_and_libraries_2019.4.243/linux/compiler/lib/intel64_lin/libintlc.so.5 (0x0000145cad78d000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x0000145cad571000)
        libc.so.6 => /lib64/libc.so.6 (0x0000145cad1a4000)
        libdl.so.2 => /lib64/libdl.so.2 (0x0000145cacfa0000)
        /lib64/ld-linux-x86-64.so.2 (0x0000145cb0365000)

So the basic build environment with the set paths and options I set
does seem to work. But, the test for ICU uses a different setup and
thus fails. Excerpt from config.log:

intel-linux.compile.c++ bin.v2/libs/regex/build/intel-linux/debug/threading-multi/visibility-hidden/has_icu_test.o

    "icpc" -c -xc++ -fvisibility-inlines-hidden -fPIC -m64 -pthread -O0 -g -fvisibility=hidden -w2 -inline-level=0  -DBOOST_ALL_NO_LIB=1 -DBOOST_HAS_ICU=1 -I"." -I"/sw/env/gcc8intel-19.0.4_impi/icu/64.2/include"   -c -o "bin.v2/libs/regex/build/intel-linux/debug/threading-multi/visibility-hidden/has_icu_test.o" "libs/regex/build/has_icu_test.cpp"

intel-linux.link bin.v2/libs/regex/build/intel-linux/debug/threading-multi/visibility-hidden/has_icu

    "icpc" -L"/sw/env/gcc8intel-19.0.4_impi/icu/64.2/bin" -L"/sw/env/gcc8intel-19.0.4_impi/icu/64.2/lib" -Wl,-R -Wl,"/sw/env/gcc8intel-19.0.4_impi/icu/64.2/bin" -Wl,-R -Wl,"/sw/env/gcc8intel-19.0.4_impi/icu/64.2/lib" -Wl,-rpath-link -Wl,"/sw/env/gcc8intel-19.0.4_impi/icu/64.2/bin" -Wl,-rpath-link -Wl,"/sw/env/gcc8intel-19.0.4_impi/icu/64.2/lib" -o "bin.v2/libs/regex/build/intel-linux/debug/threading-multi/visibility-hidden/has_icu" "bin.v2/libs/regex/build/intel-linux/debug/threading-multi/visibility-hidden/has_icu_test.o"  -lrt -licudata -licui18n -licuuc  -fPIC -m64 -pthread -g -fvisibility=hidden -fvisibility-inlines-hidden 

testing.unit-test bin.v2/libs/regex/build/intel-linux/debug/threading-multi/visibility-hidden/has_icu.passed

    LD_LIBRARY_PATH="/sw/compiler/intel-19.0.4/compilers_and_libraries_2019.4.243/linux/bin/lib:/sw/compiler/intel-19.0.4/compilers_and_libraries_2019.4.243/linux/lib/intel64:/sw/env/gcc8intel-19.0.4_impi/icu/64.2/bin:/sw/env/gcc8intel-19.0.4_impi/icu/64.2/lib:$LD_LIBRARY_PATH"
export LD_LIBRARY_PATH

     "bin.v2/libs/regex/build/intel-linux/debug/threading-multi/visibility-hidden/has_icu"  && touch  "bin.v2/libs/regex/build/intel-linux/debug/threading-multi/visibility-hidden/has_icu.passed"

bin.v2/libs/regex/build/intel-linux/debug/threading-multi/visibility-hidden/has_icu: /lib64/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by /sw/env/gcc8intel-19.0.4_impi/icu/64.2/lib/libicui18n.so.64)
bin.v2/libs/regex/build/intel-linux/debug/threading-multi/visibility-hidden/has_icu: /lib64/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by /sw/env/gcc8intel-19.0.4_impi/icu/64.2/lib/libicuuc.so.64)
...failed testing.unit-test bin.v2/libs/regex/build/intel-linux/debug/threading-multi/visibility-hidden/has_icu.passed...
...failed updating 1 target...

The build itself bas successful, but it misses the library path:

[sw_at_node002 boost_1_71_0]$ ldd bin.v2/libs/regex/build/intel-linux/debug/threading-multi/visibility-hidden/has_icu
bin.v2/libs/regex/build/intel-linux/debug/threading-multi/visibility-hidden/has_icu: /lib64/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by /sw/env/gcc8intel-19.0.4_impi/icu/64.2/lib/libicui18n.so.64)
bin.v2/libs/regex/build/intel-linux/debug/threading-multi/visibility-hidden/has_icu: /lib64/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by /sw/env/gcc8intel-19.0.4_impi/icu/64.2/lib/libicuuc.so.64)
        linux-vdso.so.1 =>  (0x00007ffc69f0d000)
        librt.so.1 => /lib64/librt.so.1 (0x0000149962ae9000)
        libicudata.so.64 => /sw/env/gcc8intel-19.0.4_impi/icu/64.2/lib/libicudata.so.64 (0x00001499610a5000)
        libicui18n.so.64 => /sw/env/gcc8intel-19.0.4_impi/icu/64.2/lib/libicui18n.so.64 (0x0000149960cfc000)
        libicuuc.so.64 => /sw/env/gcc8intel-19.0.4_impi/icu/64.2/lib/libicuuc.so.64 (0x0000149960aad000)
        libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00001499607a6000)
        libm.so.6 => /lib64/libm.so.6 (0x00001499604a4000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x000014996028e000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x0000149960072000)
        libc.so.6 => /lib64/libc.so.6 (0x000014995fca5000)
        libdl.so.2 => /lib64/libdl.so.2 (0x000014995faa1000)
        libimf.so => /sw/compiler/intel-19.0.4/compilers_and_libraries_2019.4.243/linux/compiler/lib/intel64_lin/libimf.so (0x000014995f501000)
        libsvml.so => /sw/compiler/intel-19.0.4/compilers_and_libraries_2019.4.243/linux/compiler/lib/intel64_lin/libsvml.so (0x000014995db5d000)
        libirng.so => /sw/compiler/intel-19.0.4/compilers_and_libraries_2019.4.243/linux/compiler/lib/intel64_lin/libirng.so (0x000014995d7eb000)
        libintlc.so.5 => /sw/compiler/intel-19.0.4/compilers_and_libraries_2019.4.243/linux/compiler/lib/intel64_lin/libintlc.so.5 (0x000014995d579000)
        /lib64/ld-linux-x86-64.so.2 (0x0000149962cf1000)

It picks up /lib64/libstdc++.so.6, which is not what is needed.

Specifically, I set LD_RUN_PATH in the environment. Ideally, builds
don't do anything to mess with library paths, perhaps add their own
prefix to LD_RUN_PATH, and things just work. Of course, usually you
have to turn that path list into -Wl,-rpath settings to still be
effective, since builds start adding those anyway. I am not sure about
the details of b2 regarding rpath handling, but the end result seems to
be what I want:

[sw_at_node002 boost_1_71_0]$ readelf --dynamic stage/lib/libboost_regex.so | grep RPATH
 0x000000000000000f (RPATH)              Library rpath: [/sw/env/gcc8intel-19.0.4_impi/expat/2.2.7/lib:/sw/env/gcc8intel-19.0.4_impi/icu/64.2/lib:/sw/compiler/intel-19.0.4/compilers_and_libraries_2019.4.243/linux/mpi/intel64/libfabric/lib:/sw/compiler/intel-19.0.4/compilers_and_libraries_2019.4.243/linux/mpi/intel64/lib/release:/sw/compiler/intel-19.0.4/compilers_and_libraries_2019.4.243/linux/mpi/intel64/lib:/sw/compiler/intel-19.0.4/compilers_and_libraries_2019.4.243/linux/ipp/lib/intel64:/sw/compiler/intel-19.0.4/compilers_and_libraries_2019.4.243/linux/compiler/lib/intel64_lin:/sw/compiler/intel-19.0.4/compilers_and_libraries_2019.4.243/linux/mkl/lib/intel64_lin:/sw/compiler/intel-19.0.4/compilers_and_libraries_2019.4.243/linux/tbb/lib/intel64/gcc4.7:/sw/compiler/intel-19.0.4/compilers_and_libraries_2019.4.243/linux/daal/lib/intel64_lin:/sw/compiler/intel-19.0.4/compilers_and_libraries_2019.4.243/linux/daal/../tbb/lib/intel64_lin/gcc4.4:/sw/compiler/gcc-8.3.0/lib64:/sw/compiler/gcc-8.3.0/lib]

You see a bootload of library paths, resulting from our environment
with differing prefixes, though most of them from the Intel compiler
alone. Note the /sw/compiler/gcc-8.3.0/lib at the end (thus, added
first to the environment).

The icu test is built differently and thus fails to work:

[sw_at_node002 boost_1_71_0]$ readelf --dynamic bin.v2/libs/regex/build/intel-linux/debug/threading-multi/visibility-hidden/has_icu | grep RPATH
 0x000000000000000f (RPATH)              Library rpath: [/sw/env/gcc8intel-19.0.4_impi/icu/64.2/bin:/sw/env/gcc8intel-19.0.4_impi/icu/64.2/lib]

It only has a bare minimum. Actually only the path to ICU itself, and
nothing else. How should I configure the build to use a common set of
rpaths for all builds happening in the context of configuring and
building the Boost libraries?

I do not think that my setup is so unique, but it is also not that
common to everybody: It is a HPC installation on a cluster that lives
for a certain time and promises a stable base environment for
scientists (hence, no upgrade to newer OS in between, apart from the
minor CentOS releases we have to take for security patches). So we
build lots of software outselves on top of it, sometimes with the aid
of tools like EasyBuild or pkgsrc (which failed to build Boost using
the Intel compiler for me, works fine with gcc).

I already tried --dll-path, but which does not influence the has_icu
test:

#!/bin/sh
set -e
. ../../rrz-build-env.sh
module load icu
module load expat
cd boost_1_71_0
rm -f bin.v2/project-cache.jam
test -e b2 || ./bootstrap.sh --with-toolset=intel-linux
./b2 -d 3 --hardcode-dll-paths=true --dll-path=/sw/compiler/gcc-8.3.0/lib64 --with-regex

I'd appreciate any help in getting fully-featured Boost built with
Intel+GCC here.

Regards,

Thomas

-- 
Dr. Thomas Orgis
HPC @ Universität Hamburg