Re: Linking libboost_python.so to libpython

From: Niall Douglas (s_sourceforge_at_[hidden])
Date: 2020-04-03 17:02:52

On 03/04/2020 17:17, Jonathan Wakely via Boost wrote:
> The change was made for https://bugzilla.redhat.com/show_bug.cgi?id=1102667
> 
> That links to https://svn.boost.org/trac10/ticket/2615 which suggests other
> people agree that linking to libpython is correct. Dave Abrahams seems to
> be the origin of the "linking will cause duplicate symbols" issue, which
> might only be the case when the python interpreter is statically linked to
> libpython.a. Since linux distros usually (maybe always?) dynamically link
> the python interpreter to libpythonX.Ym.so, that isn't an issue, and
> Boost's behaviour is wrong, and Fedora and RHEL are probably not the only
> distros having to kluge around it.

Dave got that originally from me.

The problem was this:

1. Python extension A is built against Boost.Python v1.X.

2. Python extension B is built against Boost.Python v1.Y.

3. What happens if end user imports Python extensions A and B into the
same Python?

If Boost.Python insists on a Python link for Extension A, and a
different Python link for Extension B, now what happens?

The generally accepted hack was to load Python extensions using
dlopen(RTLD_LOCAL), which Python accepted as the least worst of all
options back in the early 2000s, and incorporated that into CPython as
the default semantic for loading extensions. Therefore there is no
global symbol resolution at play here, instead Boost.Python ends up
loading multiple copies of libpython.so, which is for obvious reasons
very bad. So we don't do that.

I have no idea what has happened since then, and all the above
information may be out of date. But that was the rationale ~20 years ago
or so. It made a lot of sense at that time because of the fundamental
brokenness that is ELF. Note that neither PE nor MachO have any of these
problems.

Since ~2002, I patched GCC with -fvisibility in ~2005, and Dave
committed a patchset for exporting visible symbols for a good chunk of
Boost. So it may now be possible to completely change how Boost.Python
does things, and meet your request.

Niall