Re: Linking libboost_python.so to libpython

From: Jonathan Wakely (jwakely.boost_at_[hidden])
Date: 2020-04-03 16:17:02

On Fri, 3 Apr 2020 at 16:53, Stefan Seefeld <stefan_at_[hidden]> wrote:

> Hi Jonathan,
> On 2020-04-03 11:43 a.m., Jonathan Wakely via Boost wrote:
>
> tools/build/src/tools/python.jam says:
>
>     # On *nix, we do not want to link either Boost.Python or Python
> extensions
>     # to libpython, because the Python interpreter itself provides all those
>     # symbols. If we linked to libpython, we would get duplicate symbols. So
>     # declare two targets -- one for building extensions and another for
>     # embedding.
>
> Why would that give duplicate symbols? Wouldn't ELF symbol interposition
> mean that only one is used? Why would it be a problem to link to libpython?
>
> Do all *nix variants use ELF these days ?
>

All the ones in common usage, yes. And for my purposes, I only care about
one specific OS (Fedora Linux) which is definitely ELF.

> In Fedora's Boost RPM packages we *do* link libboost_python.so to the
> system libpython (and we don't get any problems).
>
> What is your rationale for this, given that (according to the above
> reasoning) symbols will be resolved from the application that's loading the
> module ?
>

The change was made for https://bugzilla.redhat.com/show_bug.cgi?id=1102667

That links to https://svn.boost.org/trac10/ticket/2615 which suggests other
people agree that linking to libpython is correct. Dave Abrahams seems to
be the origin of the "linking will cause duplicate symbols" issue, which
might only be the case when the python interpreter is statically linked to
libpython.a. Since linux distros usually (maybe always?) dynamically link
the python interpreter to libpythonX.Ym.so, that isn't an issue, and
Boost's behaviour is wrong, and Fedora and RHEL are probably not the only
distros having to kluge around it.

 But to do this, we have
> these two local patches:
> https://src.fedoraproject.org/rpms/boost/blob/master/f/boost-1.57.0-python-libpython_dep.patch
>
> and then to make it link to libpython3.7m.so (rather than libpython3.7.so)
> we have this patch to pass the "m" into the Boost build:
> https://src.fedoraproject.org/rpms/boost/blob/master/f/boost-1.66.0-python-abi_letters.patch
>
> I'd prefer not to need these local hacks. Would the upstream Boost.Python
> consider making changes so that libboost_python.so can be (optionally)
> linked to libpython?
>
> Is there a cleaner way to do this than those patches?
>
> I fully agree that whatever seems the right choice of build (including
> link) process for a given platform, we (boost) should incorporate that into
> our own infrastructure, so downstream package maintainers don't need to
> patch it. I'll leave it to the Boost.Build maintainers to answer the
> technical details of that process.
>

I should note that the linked patches worked fine up to Boost 1.69.0 but
since 1.70.0 they break the build somehow, so that the mpi.so python
extensions are no longer built. I am unable to figure out the problem,
because I find jamfiles completely impenetrable. I'd love to try using
cmake instead, but that doesn't seem to be supported for Boost.MPI.

Boost 1.70.0 and 1.71.0 fail to build with:

...updating 24 targets...
common.copy /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_graph_parallel.so.1.71.0
common.copy /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_wserialization.so.1.71.0
common.copy /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_mpi.so.1.71.0
common.copy /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_serialization.so.1.71.0
ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_graph_parallel.so
common.copy /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_graph_parallel.a
ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_wserialization.so
ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_serialization.so
ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_mpi.so
ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_serialization.so.1
ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_serialization.so.1.71
ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_mpi.so.1.71
ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_mpi.so.1
common.copy /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_wserialization.a
ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_wserialization.so.1
common.copy /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_serialization.a
ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_wserialization.so.1.71
ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_graph_parallel.so.1
ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_graph_parallel.so.1.71
common.copy /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_mpi.a
...updated 24 targets...
+ mkdir -p /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/python3.8/site-packages/openmpi/boost
+ touch /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/python3.8/site-packages/openmpi/boost/__init__.py
+ mv /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/mpi.so
/builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/python3.8/site-packages/openmpi/boost/
mv: cannot stat
'/builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/mpi.so':
No such file or directory

And 1.72.0 fails even earlier, with what looks like a conflict between
building Boost.Serialization with "python" and "python-for-extensions"
properties (if I'm reading python.jam right, which is unlikely):

+ echo ============================= build openmpi-x86_64 ==================
+ ./b2 -d+2 -q -j8 --with-mpi --with-graph_parallel
--build-dir=openmpi-x86_64 variant=release threading=multi
debug-symbols=on pch=off python=3.8 stage
Performing configuration checks

    - default address-model    : 64-bit
    - default architecture     : x86
warning: non-free usage requirements <threading>multi ignored
warning: in main-target mpi at
/builddir/build/BUILD/boost_1_72_0/tools/build/src/user-config.jam:6
warning: non-free usage requirements <threading>multi ignored
warning: in main-target boost_mpi at libs/mpi/build/Jamfile.v2:120
warning: non-free usage requirements <threading>multi ignored
warning: in main-target boost_mpi_python at libs/mpi/build/Jamfile.v2:145
error: Name clash for
'<p/builddir/build/BUILD/boost_1_72_0/stage/lib>libboost_serialization.so.1.72.0'
error:
error: Tried to build the target twice, with property sets having
error: these incompatible properties:
error:
error:     -  none
error:     -  <dll-path>/usr/lib <dll-path>/usr/lib/python3.8/config
error:
error: Please make sure to have consistent requirements for these
error: properties everywhere in your project, especially for install
error: targets.