[Boost-build] error building Boost.Python for multiple Python versions.

Subject: [Boost-build] error building Boost.Python for multiple Python versions.
From: Stefan Seefeld (stefan_at_[hidden])
Date: 2017-12-08 14:19:45

Hello,

looking into https://github.com/boostorg/python/issues/129, I think I
understand the underlying issue, which, I think, is a Boost.Build issue.
After adding

using python : 2.7 ;
using python : 3.5 : python3 : /usr/include/python3.5m :
/usr/lib64/libpython3.5m.so ;

to my user-config.jam file, and running `./b2 --with-python -d2`, I
observe the following output:

...

gcc.compile.c++
bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/list.o

    "g++"   -O3 -finline-functions -Wno-inline -Wall -pthread -m64 
-DBOOST_ALL_NO_LIB=1 -DBOOST_PYTHON_SOURCE -DBOOST_PYTHON_STATIC_LIB
-DNDEBUGÂ  -I"." -I"/usr/include/python2.7" -c -o
"bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/list.o"
"libs/python/src/list.cpp"
...

gcc.archive
bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/libboost_python.a

    "ar"  rc
"bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/libboost_python.a"
"bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/list.o"
"bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/long.o"
"bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/dict.o"
"bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/tuple.o"
"bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/str.o"
"bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/slice.o"
"bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/converter/from_python.o"
"bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/converter/registry.o"
"bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/converter/type_id.o"
"bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/object/enum.o"
"bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/object/class.o"
"bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/object/function.o"
"bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/object/inheritance.o"
"bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/object/life_support.o"
"bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/object/pickle_support.o"
"bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/errors.o"
"bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/module.o"
"bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/converter/builtin_converters.o"
"bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/converter/arg_to_python_base.o"
"bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/object/iterator.o"
"bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/object/stl_iterator.o"
"bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/object_protocol.o"
"bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/object_operators.o"
"bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/wrapper.o"
"bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/import.o"
"bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/exec.o"
"bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/object/function_doc_signature.o"

...

gcc.archive
bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/libboost_python3.a

    "ar"  rc
"bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/libboost_python3.a"
"bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/list.o"
"bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/long.o"
"bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/dict.o"
"bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/tuple.o"
"bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/str.o"
"bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/slice.o"
"bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/converter/from_python.o"
"bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/converter/registry.o"
"bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/converter/type_id.o"
"bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/object/enum.o"
"bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/object/class.o"
"bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/object/function.o"
"bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/object/inheritance.o"
"bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/object/life_support.o"
"bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/object/pickle_support.o"
"bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/errors.o"
"bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/module.o"
"bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/converter/builtin_converters.o"
"bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/converter/arg_to_python_base.o"
"bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/object/iterator.o"
"bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/object/stl_iterator.o"
"bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/object_protocol.o"
"bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/object_operators.o"
"bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/wrapper.o"
"bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/import.o"
"bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/exec.o"
"bin.v2/libs/python/build/gcc-gnu-7/release/link-static/threadapi-pthread/threading-multi/object/function_doc_signature.o"
...

Notice how the "link.cpp" source file (tracing an arbitrary source file
from Boost.Python) gets compiled once (against Python 2.7 headers), then
linked twice, once for libboost_python (i.e. Python 2.7), and once for
libboost_python3 (i.e. Python 3.5).

This is clearly wrong. If Boost.Python is compiled for multiple Python
versions in the same process, the build tree needs to differentiate
between the intermediate object files the same way it differentiates
between objects compiled with different toolchains. Instead, Boost.Build
gets confused and links two Boost.Python libraries from the same object
files.

I'm normally not building for multiple Python versions in the same
process (and neither for multiple toolchains), but as the script that
generates Windows binaries uses that process, the generated binaries are
wrong.

What needs to happen to fix this ? My personal advice would be to build
for one Python version at a time (even if the resulting binaries get
bundled into the same package). To support what some believe to be an
important use-case, the internal Boost.Build logic needs to be augmented
to add a <python-version> component to the build path, to make sure
objects get rebuild properly for each additional Python version.

Thoughts ?

Stefan

-- 
      ...ich hab' noch einen Koffer in Berlin...