Python and pointer questions

From: david.abrahams_at_[hidden]
Date: 2001-09-13 10:27:48

> Hi,
> I am new to this group,
> i am about to write yet another C++ API interface generator for 
python
> (there are
> still too few out there!)

Could you explain what you mean by "interface generator"?

> However, i found i needed an additional conversion to/from unsigned
> character
> strings.
> Could these be added to boost::python (i am using boost 1.23.0) ?

<snip>

These seem reasonable to me. It is not entirely clear to me that the
correct conversion to const char* tries to interpret the Python object
as a String, but I can't think of any other way to accomodate const
char*... so I vote yes.

> 
> also, i found it more useful to pass a null pointer to a function 
expecting
> a const char*, if
> specifying None as an argument (see above):
> 
> const char* from_python(PyObject* p, boost::python::type<const 
char*>)
> {
>   if (p == Py_None) {
>     return (const char*)0;
>   } else {
>     const char* s = PyString_AsString(p);
>     if (!s)
>         throw boost::python::argument_error();
>     return s;
>   }
> }
>
> could the part in conversions.cpp be replaced by this ?

This is a bit more controversial. None is not a string, and it is not
clear to me that it should be treated as one. If you pass None to a
function expecting a string in Python, then try to operate on it as
though it were a string, you'll get a runtime error. Often, C++
functions which take char*/const char* expect the arguments to point
at valid, if empty, null-terminated byte strings. I could be convinced
that your approach is the right one, but I would be more easily
persuaded to return a pointer to a valid, empty string:

const char* from_python(PyObject* p, boost::python::type<const char*>)
{
  if (p == Py_None) {
    return (const char*)"";
  } else {
    const char* s = PyString_AsString(p);
    if (!s)
        throw boost::python::argument_error();
    return s;
  }
}

> Also, i don't have much of an idea how to wrap pointers to C++ class
> objects in a way such that
> i don't have to wrap the whole class in boost python. This is 
especially
> needed for wrapping pointers
> to opaque structs, like the XWindows Display data type. Currently i 
am
> using this:
> 
> struct _XDisplay {
> };
> 
> typedef struct _XDisplay Display;
> 
> PyObject* to_python(struct _XDisplay* p)
> {
>   return PyCObject_FromVoidPtr(p,NULL);
> }
> 
> struct _XDisplay* from_python(PyObject* p, python::type<struct
> _XDisplay*const&>)
> {
>   return reinterpret_cast<struct _XDisplay*>(PyCObject_AsVoidPtr
(p));
> }
> 
> which doesn't look like an exactly clean solution to me. any ideas ?

Hmm. I understand your problem. Boost.Python doesn't supply automatic
pointer conversions, because in general they are unsafe. The theory is
that Python should be a safe environment where crashes aren't
possible. Opaque pointers, OTOH, should probably be treated as an
exception to the rule, especially where a checked API is being used to
operate on them.

I have an idea, but it requires partial ordering of function
templates. If you need MSVC support you will be out-of-luck. On the
other hand, a sufficiently inventive person might be able to find a
way to make that work as well:

template <class T> struct opaque_pointer;

template <class T>
PyObject* to_python(T* p)
{
  BOOST_STATIC_ASSERT(opaque_pointer<T>::value);
  return PyCObject_FromVoidPtr(p,NULL);
}

template <class T>
T* from_python(PyObject* p, python::type<T*const&>)
{
  BOOST_STATIC_ASSERT(opaque_pointer<T>::value);
  return reinterpret_cast<T*>(PyCObject_AsVoidPtr(p));
}

Then you can specialize opaque_pointer<T> for your types:

template <>
struct opaque_pointer<_XDisplay>
{
     BOOST_STATIC_CONSTANT(bool, value = true);
};

> Lastly, in the API i am trying to wrap, const and non-const 
pointers to a
> wrapped
> class are returned as method arguments all the time.
> The boost documentation (libs/python/doc/pointers.html) proposes:
> 
> BOOST_PYTHON_BEGIN_CONVERSION_NAMESPACE // this is a gcc 2.95.2 bug
> workaround
>   PyObject* to_python(Foo* p)
>   {
>       return
> boost::python::python_extension_class_converters<Foo>::ptr_to_python
(p);
>   }
> 
>   PyObject* to_python(const Foo* p)
>   {
>       return to_python(const_cast<Foo*>(p));
>   }
> BOOST_PYTHON_END_CONVERSION_NAMESPACE
> 
> i didn't see any ptr_to_python method, i assume smart_ptr_to_python 
was
> meant.

Yes.

> However, i didn't find documentation and didn't understand 
everything on
> this.
> Could anyone clarify this to me ?

What this does, when you return a pointer to Foo, is to create a NEW
python object which wraps the "smart pointer" type you have passed
(which in this case is a raw pointer). Obviously, this is unsafe,
because the pointer now refers to your object which lives inside
another Python object that can be destroyed when its reference count
goes to zero. We have discussed various solutions for allowing users
access to the actual Python object wrapping a C++ object, but none of
them have been implemented.

> In the library i am trying to wrap (Trolltech's Qt)
> trees of QObject (the class i have wrapped) are built. The problem 
is, in
> some cases,
> these objects are instantiated both by the API or by me (from 
python) and
> destroyed both
> by me or by the API. I made a subclass of QObject (as proposed for
> overriding virtual
> methods in python), this also works so far.

Does Qt have some kind of reference-counting mechanism to manage its
objects? If so, you can create a specialized smart pointer object for
use with Qt objects, and arrange for it to be convertible to Python
using smart_ptr_to_python. That way, your objects will always be
properly and safely managed.

> Both Qt-created objects and python-created objects may be returned
> by Qt methods. Of course i need to somehow keep track of these 
objects.
> My question is, how is an object created and returned by the C++ API
> wrapped into a python
> object ?

If it's returned by value or by const&, a /copy/ is made (using the
copy-constructor) and placed within storage managed by the resulting
Python object. If it is returned by smart pointer, a copy of the smart
pointer is made and placed within the Python object's storage. 

> And how can i make sure that a particular C++ object returned by the
> API is identified by a unique Python object (such that i can compare
> them by the python 'is' operator) ?  Is there an 'object registry'
> or similar ?

There is currently no database which links C++ object addresses to the
corresponding Python objects. That might be possible, but there are
several issues:

1. Not all users should have to pay for this tracking

2. It might be impossible to do correctly in cases of multiple
inheritance, since the addresses of base subobjects are not
neccessarily identical. 

If you are interested in trying to address some of these issues, I'm
certain your help with Boost.Python would be appreciated. AFAIK, Ralf
Grosse-Kunstleve is gathering interested parties to work on some
planned revisions.

-Dave