Re: [boost] Python and pointer questions

From: David Abrahams (david.abrahams_at_[hidden])
Date: 2001-09-14 10:21:19

> On Thu, 13 Sep 2001, you wrote:
> >
> > Could you explain what you mean by "interface generator"?
> >
> 
> A program being fed C++ header files as input and generating code
> which implements a python module as output, the python module
> implements the C++ classes found in the input header files.  The
> parser is based on openc++ (http://www.is.titech.ac.jp/~chiba), with
> some modifications.  I am using this for generating a python module
> wrapping the Qt/KDE classes.  This has already been done by Phil
> Thompson (see his pages on theKompany), but in a (my opinion) rather
> crude, non-C++ way. Currently, he has much of a struggle supporting
> all Qt/KDE versions (beacuse he has to reiterate the whole API
> declarations in 'interface files'), also the build process is not
> exactly easy. My approach bases on the orginal, unmodified C++ header
> files, which are parsed by a full-blown C++ parser.

OK, great. Last time we looked at openc++, it wasn't doing the whole
parsing job, though. Just be aware that unless things have changed you
may find corners of C++ which it can't handle. Either way, I think
your front-end could be an incredibly valuable addition to
Boost.Python.

> > > could the part in conversions.cpp be replaced by this ?
> >
> > This is a bit more controversial. None is not a string, and it is not
> > clear to me that it should be treated as one. If you pass None to a
> > function expecting a string in Python, then try to operate on it as
> > though it were a string, you'll get a runtime error. Often, C++
> > functions which take char*/const char* expect the arguments to point
> > at valid, if empty, null-terminated byte strings. I could be convinced
> > that your approach is the right one, but I would be more easily
> > persuaded to return a pointer to a valid, empty string:
> >
> ...
> >
> 
> If i needed to pass an zero-length string to the function/method, i
> would do so from python. But many C++ functions actually expect a NULL
> char*, meaning 'argument not specified, create a default one'. Most
> notably are constructors with a name argument, which, when not
> specified, should create a name automatically. This is used throughout
> the Qt API.

Understood. Perhaps this should be an optional behavior. I don't
know. Right now, I'm not able to use Boost.Python for my work so I
don't have as much of a vested interest as the other users. I think
you should try to raise a consensus among the Boost.Python user
community.

> > > Also, i don't have much of an idea how to wrap pointers to C++
> > > class objects in a way such that i don't have to wrap the whole
> > > class in boost python. This is especially needed for wrapping
> > > pointers to opaque structs, like the XWindows Display data
> > > type. Currently i ...
> > >
> > > which doesn't look like an exactly clean solution to me. any ideas ?
> >
> > Hmm. I understand your problem. Boost.Python doesn't supply automatic
> ...
> > Then you can specialize opaque_pointer<T> for your types:
> >
> > template <>
> > struct opaque_pointer<_XDisplay>
> > {
> >      BOOST_STATIC_CONSTANT(bool, value = true);
> > };
> >
> 
> interesting. But what i was thinking of is something like a template
> class derived from PyTypeObject with instances holding just a pointer
> to the opaque type

Why derive it from PyTypeObject? I think you just want an object, not
a Python type...

> , with no (python-wise) constructor or any methods
> provided. The instances may only be created by function
> return/call-by-reference conversions and passed to some other
> functions ( by means of from_python/to_python), but still would
> provide for some type-safety (such that you can not pass a XDisplay*
> where a XEvent* was expected.  

Sure, this makes sense. But I think my proposal above will allow you
to do most of that.

> It was somewhat difficult for me to
> understand the classes found in boost/python/types.hpp and
> classes.hpp, but now i have at least some basic knowledge of them, so
> maybe i'm going to try a thing like this by myself.

I would encourage you to reuse more of the high-level machinery in
Boost.Python. It seems like you keep writing low-level Python types
instead of taking advantage of the wrapping facilities. They could
make your life easier.

> > What this does, when you return a pointer to Foo, is to create a NEW
> > python object which wraps the "smart pointer" type you have passed
> > (which in this case is a raw pointer). Obviously, this is unsafe,
> > because the pointer now refers to your object which lives inside
> > another Python object
> 
> not necessarily - it could have been created either by the wrapped C++
> API or by python. 

Either way, it's lifetime is not managed appropriately by Python so
that the pointer will be valid as long as the returned object is
alive. That is the sense in which I say it's unsafe.

> Currently, i am using a callback subclass (storing a
> reference to the corresponding python object) and a dynamic_cast in
> to_python to find out whether the returned object was created by
> python or solely by the API.  

Sure, that works. I'd probably have suggested it, but I've been away
from the code for some time.

> Concerning the Qt API, structured
> widgets (like a file dialog) can consist of Qt-generated objects, or
> by user-created widgets, or a mix thereof - so i must be able to
> distinguish between them.

It sounds like you have hit on a reasonable approach.

> > Does Qt have some kind of reference-counting mechanism to manage its
> > objects? If so, you can create a specialized smart pointer object for
> 
> No, unfortunately not.

Too bad.

> > > And how can i make sure that a particular C++ object returned by the
> > > API is identified by a unique Python object (such that i can compare
> > > them by the python 'is' operator) ?  Is there an 'object registry'
> > > or similar ?
> >
> > There is currently no database which links C++ object addresses to the
> > corresponding Python objects. That might be possible, but there are
> > several issues:
> >
> > 1. Not all users should have to pay for this tracking
> 
> That's right, it should really be an optional part.  Having to do cope
> with pointers to non-wrapped classes, it would still be useful.

Well, your dynamic_cast approach seems to work, as long as everything
has a virtual function.

> > 2. It might be impossible to do correctly in cases of multiple
> > inheritance, since the addresses of base subobjects are not
> > neccessarily identical.
> 
> I can't imagine storing base subobjects would be of relevance, but
> maybe that's just my lack of imagination.  Normally, API-returned
> object pointers will be used 'as-is', without further downcasting. If
> downcasting from a (multiply used) base class were needed to pass the
> pointer into another API function, that would be hard (impossible?) to
> do from python anyway, so i believe this should not be of concern. I
> do believe it would be sufficient to register pointers to the topmost
> base class and mapping them to python objects.  In to_python, one
> would upcast to this base class and search for the pointer (after
> having checked it's not a python-created callback subclass object)
> returning the associated python object if found, and register and
> create a new python object if not. If the reference to the python
> object is dropped, it's just removed from the object database.

But there is no common topmost base class :(

>  The
> difficulty is when the underlying C++ object is deleted while still
> being referenced from python objects, but as i understood this problem
> is currently present anyway.

No, that's part of what I've been trying to stress. The system is
designed so that you can't create such a situation without resorting
to some non-sanctioned shenanigans.

-Dave