[Boost-bugs] [Boost C++ Libraries] #10799: Several problems building the example/tutorial project

Subject: [Boost-bugs] [Boost C++ Libraries] #10799: Several problems building the example/tutorial project
From: Boost C++ Libraries (noreply_at_[hidden])
Date: 2014-11-17 22:21:29

#10799: Several problems building the example/tutorial project
 Reporter: m@… | Owner: rwgk
     Type: Bugs | Status: new
Milestone: To Be Determined | Component: Python
  Version: Boost 1.57.0 | Severity: Showstopper
 Keywords: |
 -- There are a number of problems reported here, so you may want to break
 this down into pieces for separate tracking. However, I would consider
 this item to be fixed only when I can build the example/tutorial and it

 I had a hard time building the example/tutorial project. Some of this is,
 I believe, due to bugs in the distribution, and some is probably my lack
 of understanding.

 Some of this I posted to the Boost interest group before I could join this
 (Python C++) SIG, hoping it would get forwarded on here, so there may be a

 1. Calling bjam in the example/tutorial project failed to even start up
 the build system. This was because the example/bootstrap.jam pointed to
 the wrong path for the build system root. Instead of
 ../../../tools/build/v2, it should be ../../../tools/build/src/kernel.
 When I changed this, bjam now got past the build system startup.

 2. Building the project not only compiles hello.cpp, but it also builds a
 private copy of the Boost/Python library. So I needed to supply the
 properties needed to correctly build this library (i.e., link=shared,
 address-mode=64). Of course, I needed to supply those same properties
 anyway as part of creating the extension.

 There's probably a way to change something so that the extension uses the
 library built in the Boost.Python's own project, or if I have obtained the
 libraries without having to build them, it would use these. I don't know
 if you intended to have the tutorial example make its own copies, but it
 seems a waste of resources to do so.

 3. The link for debug mode failed, saying that the .pdb file was in the
 wrong format (LNK1207). This is a bug, due to an option in the link
 command '/IMPORTLIB:...hello_ext.pdb'. So the linker is creating the .pdb
 file as an import library, then complaining that it's not a valid pdb
 file. I changed '.pdb' to '.lib'. I could also have removed this option
 entirely, since hello_ext.pyd doesn't have anything to export anyway.

 4. Before figuring out that the link was the problem, I changed the /Z7
 argument to /Zi in the compile command for hello.cpp. I don't know if
 this was necessary, or if it was necessary to leave it in place. For now,
 I just wanted to get it to build. Without /Z7, the debug symbols go into
 example/tutorial/vc120.pdb. I don't know if the linker found these or
 not. When I try stepping into the extension, I'll know for sure.
 Microsoft prefers that .pdb files be used for debug symbols rather than
 embedding them in the .obj files, so this might be the only real reason to
 make the change.

 5. The link for both release mode failed with two undefined symbols,
 __imp_DecodePointer and __imp_EncodePointer, which are in kernel32.lib. I
 tried adding kernel32.lib to the link's inputs. But then it warned that
 there could be static constructors/destructors not getting called. After
 much research on this topic, I found that the source of the problem was
 the /NOENTRY argument in the link command, which has the consequence that
 the automatic CRT initialization of the DLL doesn't occur. So I remove
 the /NOENTRY and got not warnings, and I didn't need to add kernerl32.lib

 6. A minor point. The MACHINE:X64 is redundant. The linker knows it's
 X64 because of the architecture of the input files and libraries. Nothing
 wrong with it being there, but it's just clutter in the code.

 7. Now bjam was successful in building whatever it wanted to build. It
 said that hello.test passed. Sounds great, I thought. But I then went
 into Python and tried 'import hello_ext' and that failed. So I have an
 issue with the test program passing, while the extension didn't actually

 8. The problem was that bjam didn't put the hello_ext.pyd file in my
 Python's lib/site-packages folder. It built the .pyd and .pdb files in
 the example/tutorial/bin/... staging area, and copied (ONLY) the .pyd file
 to example/tutorial. So not only did the .pyd file get put in the wrong
 place, but the .pdb was left out. If I am going to debug my extension
 (such as with Visual Studio's Python Tools), the .pdb file also needs to
 be in the lib/site-packages folder. Without the .pdb file, the PTVS
 debugger will not set breakpoints or step into the extension code.

 9. I spent another few hours figuring out why the import statement in
 Python didn't work. Python could not load the hello_ext.pyd because it
 has references to BOOST_PYTHON_....DLL. If I run Python from the
 example/tutorial directory itself, it works because this DLL had been
 build there. That explains why the hello.test passed (as bjam ran it from
 that same directory). So to fix this problem, I copied the DLL (and its
 accompanying PDB) from the place where it was built into a directory in my
 PATH. It would also work to have these files placed in the Python lib
 /site-packages folder, since the main DLL's folder is one of the places
 that Windows looks for imported DLLs.

 To summarize, the fixes I made in order to build the example and use it in
 Python were:
     * Change the build system path in examples/bootstrap.jam.
     * Change the /IMPORTLIB:....pdb to ....lib in the link command. I
 could have removed it altogether.
     * Remove /NOENTRY from the link command.
     * Manually copy the example/tutorial/bin/.../hello_ext.(pyd,pdb) files
 to Python's lib/site-packages.
     * Manually copy the Boost.Python DLL and PDB files to a location in my
 %PATH% (or to lib/site-packages)

 These last two steps are not necessary if I run Python from the
 example/tutorial directory itself, as Python will find the PYD there and
 it will find the Boost.Python DLL there as well. So if the other problems
 are fixed, then the tutorial would be telling the truth that the extension
 can be imported and works correctly PROVIDED that you don't change
 directories after doing the build.

 Maybe there's something later in the tutorial about modifying the jamfile
 to specify where you want the files installed so that Python can import
 them no matter what directory Python was called from. If this is the
 case, that's OK with me, as long as the tutorial says '''clearly''' where
 you first build the example that at this point, Python '''must be run from
 the example/tutorial directory'''.

 This cost me about a whole day's time, and I'm pretty resourceful. And I
 actually want to build my own extensions in VS instead of bjam, linking to
 the Boost.Python DLL. I started of by trying this, and my extension
 crashed. So that's why I went about building the example with bjam, to at
 least see what a successful deployment looked like. I now understand that
 my original attempt crashed because linking to the Boost.Python DLL is not
 enough to be sure that the target PYD will load.

 If I had understood that in the first place, I would have taken care to
 put the Boost.Python DLL on my PATH, and not tried building the example in
 the manner specified in the tutorial. But then I would not have
 discovered those build bugs, and they would have gone unreported.

 I suppose that a lot of people would have given up building the tutorial
 example and then given up on Boost.Python altogether.

 One more comment. You should make sure, as part of the QA process for
 releasing Boost that you be sure that all the examples can be built and
 work properly, particularly on Windows with all the MSVC toolsets. Item
 (1) would apply to all platforms, so I would guess that the
 example/tutorial was not tested at all.

 Michael Rolle

Ticket URL: <https://svn.boost.org/trac/boost/ticket/10799>
Boost C++ Libraries <http://www.boost.org/>
Boost provides free peer-reviewed portable C++ source libraries.

This archive was generated by hypermail 2.1.7 : 2017-02-16 18:50:17 UTC