[Python] Improved exception handling for Python embedding
Hey guys! Boost.Python throws boost::python::error_already_set exceptions whenever calls to embedding functions such as boost::python::import() or boost::python::exec() fail. This exception is just an empty class. It does not provide any information on the error which occurred on the Python side. It is not even a standard exception in the C++ sense. As a result, debugging problems with embedded Python code or even the Python environment (available packages etc.) with the current exception handling mechanism is difficult. So far, in our company we wrap most Boost.Python functions with custom exception handling code. In our exception handling code, we basically work the CPython API to extract details of Python exceptions such as * Exception types, * Error messages, and * Stack traces. We want to put this part of our embedding library back into the Boost.Python project. At the EuroPython 2014 conference in Berlin, I had a few discussions with Reinhard Wobst, Austin Bingham, and others, and the current exception handling mechanism was perceived an impediment on the usability of Boost.Python. I am most grateful for feedback of seasoned Boost contributors (and users) whether my proposal is of sufficient interest and how best to pursue it. Cheers Michael -- Dr. Michael König Software Engineer Blue Yonder GmbH Ohiostraße 8 76149 Karlsruhe Tel +49 (0) 721 383 117 38 Fax +49 (0) 721 383 117 69 michael.koenig@blue-yonder.com www.blue-yonder.com Registergericht Mannheim, HRB 704547 USt-IdNr. DE 277 091 535 Geschäftsführer: Jochen Bossert, Uwe Weiss (CEO)
Hi Michael, [I believe this mail should best be sent to the c++-sig@python.org ML, so I'm cross-posting there for follow-up.] On 2014-08-01 07:40, Michael Koenig wrote:
Hey guys!
Boost.Python throws boost::python::error_already_set exceptions whenever calls to embedding functions such as boost::python::import() or boost::python::exec() fail.
Right.
This exception is just an empty class. It does not provide any information on the error which occurred on the Python side. It is not even a standard exception in the C++ sense.
As a result, debugging problems with embedded Python code or even the Python environment (available packages etc.) with the current exception handling mechanism is difficult.
I have long wanted to improve the handling of Python exceptions in Boost.Python, and wrapping the CPython exception handling API seems a good start.
So far, in our company we wrap most Boost.Python functions with custom exception handling code.
In our exception handling code, we basically work the CPython API to extract details of Python exceptions such as * Exception types, * Error messages, and * Stack traces.
Right, that seems natural and convenient. However, I'm not sure this addresses any of the points you raise above. Notably, does your API allow you to catch specific (Python) exceptions ? In other words, do you provide a C++ class hierarchy of exceptions reflecting the Python exception class hierarchy ? How do these exception types relate to the error_already_set class ?
We want to put this part of our embedding library back into the Boost.Python project. At the EuroPython 2014 conference in Berlin, I had a few discussions with Reinhard Wobst, Austin Bingham, and others, and the current exception handling mechanism was perceived an impediment on the usability of Boost.Python.
I am most grateful for feedback of seasoned Boost contributors (and users) whether my proposal is of sufficient interest and how best to pursue it.
I'm very much interested in seeing the API changes you propose, either in form of a API document or a patch. Best, Stefan -- ...ich hab' noch einen Koffer in Berlin...
On Fri, Aug 1, 2014 at 4:40 AM, Michael Koenig < michael.koenig@blue-yonder.com> wrote:
Hey guys!
Boost.Python throws boost::python::error_already_set exceptions whenever calls to embedding functions such as boost::python::import() or boost::python::exec() fail.
This exception is just an empty class. It does not provide any information on the error which occurred on the Python side. It is not even a standard exception in the C++ sense.
I took a quick look -- if I read the code right, the Boost Python exception types don't even derive from std::exception. I can't think of any reason why that's done, you're right that it should be fixed. Secondly, I suggest using BOOST_THROW_EXCEPTION to throw. This would allow arbitrary information to be attached to the exception object, not only at the throw site but also at a later time. See http://www.boost.org/doc/libs/release/libs/exception/doc/tutorial_transporti... . Emil Dotchevski Reverge Studios, Inc. http://www.revergestudios.com/reblog/index.php?n=ReCode
On 1 Aug 2014 at 12:27, Emil Dotchevski wrote:
Boost.Python throws boost::python::error_already_set exceptions whenever calls to embedding functions such as boost::python::import() or boost::python::exec() fail.
This exception is just an empty class. It does not provide any information on the error which occurred on the Python side. It is not even a standard exception in the C++ sense.
This is by design. It very intentionally does NOT inherit from std::exception in order to avoid a RTTI lookup.
I took a quick look -- if I read the code right, the Boost Python exception types don't even derive from std::exception. I can't think of any reason why that's done, you're right that it should be fixed.
Secondly, I suggest using BOOST_THROW_EXCEPTION to throw. This would allow arbitrary information to be attached to the exception object, not only at the throw site but also at a later time. See http://www.boost.org/doc/libs/release/libs/exception/doc/tutorial_transporti...
With respect to the above as it was clearly meant to be helpful, this is the wrong place to ask for help with Boost.Python - very few here know anything about that library now that Dave has left. Go to the Python C++ SIG mailing list (https://www.python.org/community/sigs/current/cplusplus-sig) where you'll get accurate help. There is about one person there nowadays who replies, but he'll reply with experience. I'd also strongly suggest that you search the archives of that mailing list. It contains a ton of very useful info on why Dave made the design choices he did. Almost all of them were correct - Dave is an outstanding engineer. (BTW error_already_set means "go ask the python runtime for the error". It is very deliberately and intentionally held separate from the C++ exception system for lots of very good reasons, some of which are summarised at http://misspent.wordpress.com/2009/10/11/boost-python-and-handling-pyt hon-exceptions/ and don't include the fun of dealing with C++ => Python => C++ => Python => C++ exception handling, but also because a lot of the time you really don't want python exceptions to propagate into C++, and an empty state non-polymorphic type exception is cheap to catch and throw away for those scenarios, plus if you do wish to extract the python exception state you doing it by hand is going to be no slower than BPL doing it for you, and you can code around the gotchas like exception throws during module loads etc). Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/
On Fri, Aug 1, 2014 at 2:21 PM, Niall Douglas <s_sourceforge@nedprod.com> wrote:
(BTW error_already_set means "go ask the python runtime for the error". It is very deliberately and intentionally held separate from the C++ exception system for lots of very good reasons
Yes, this makes perfect sense. Since the Python runtime can be used to gather anything and everything related to the failure, there is no reason to use an alternative system to transport that data to the catch site. Thanks for the correction. Emil Dotchevski Reverge Studios, Inc. http://www.revergestudios.com/reblog/index.php?n=ReCode
On 1 Aug 2014 at 15:34, Emil Dotchevski wrote:
(BTW error_already_set means "go ask the python runtime for the error". It is very deliberately and intentionally held separate from the C++ exception system for lots of very good reasons
Yes, this makes perfect sense. Since the Python runtime can be used to gather anything and everything related to the failure, there is no reason to use an alternative system to transport that data to the catch site.
Thanks for the correction.
It's only from bitter experience. My knowledge of BPL has faded enough I can no longer reliably help people with it :(. However BPL was how I first entered the world of Boost back 2001. If you dig through the c++-sig mail archives, you'll find many very angry posts by me directed against poor Dave who took it all with exceedingly good humour. I've always held him in the highest of esteems since. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/
Hey guys! Thanks for the feedback. What I gather is: 1) Yes, there is interest in decent exception handling 2) Still, it should be possible to use the nearly free exception system that it is currently in place 3) It would be nice to know which python exception has been thrown, though. For Reinhard, Austin, and me this means that we will work on additional features which allow easy access to python exceptions should the user desire. This would heighten the level of abstraction provided by Boost.Python as a whole, since less CPython API code must be written by its users. We will commit some code once we are confident we meet all requirements. Cheers Michael
participants (4)
-
Emil Dotchevski -
Michael Koenig -
Niall Douglas -
Stefan Seefeld