[boost] Re: BPL: Accessing PyObject pointer from C++

From: John Barnard (barnard_at_[hidden])
Date: 2001-01-06 00:47:55

Dave,

Dave Abrahams writes:
 > John,
 > 
 > Sorry about the delay in responding to this; for some reason I never 
 > got the mail message. I only saw it by chance when browsing boost 
 > messages on the web!

I wonder if I'm doing something wrong -- this is the second time
you've said that you didn't receive my message (I sent an unrelated
message last month). I sent both mailings to boost_at_egroups.com.

 > I'm no expert on NumPy, but I would have thought that it had its own 
 > data representation. Is it possible to create a NumPy array that uses 
 > some external block of data? Even if this is possible, you run the 
 > risk of crashing the Python interpreter if the NumPy array is used 
 > after the DVArray is destroyed.

Yes, it is possible to create a NumPy array that uses some external
block of data, in this case, the data of valarray. If I increase the
reference count to the DVArray instance when I create the NumPy array
instance, the DVArray instance should exist as long as the NumPy array
instance exists.

 > >     if (arr == NULL) return NULL;
 > >     arr->flags |= OWN_DATA;
 > 
 > Does the above line indicate to the NumPy array that it owns its 
 > data? If so, you are lying to it ;-) The data is still owned by the 
 > valarray.

No. The flag indicates that the NumPy array doesn't own the data and
shouldn't free the memory when the NumPy array is destroyed.

 > > 
 > > How do I access the reference count of the wrapped DVArray instance?
 > 
 > In what context?
 > 
 > > I
 > > would also like to gain access to the PyObject* of the wrapped
 > > instance so I can make it the base for the NumPy array instance (and
 > > subsequently have the reference count of the wrapped DVArray 
 > decrease
 > > when the NumPy array is destroyed). 
 > 
 > Ah; I think I'm beginning to understand. You want the NumPy array to 
 > manage the lifetime of the DVArray, so that the former holds a 
 > reference count to the latter as long as it exists?

Not quite. I want to use the NumPy array interface as a convenient way
to manipulate in Python the data in DVArray. As I mentioned above,
NumPy allows you to create a NumPy array that doesn't own the
data. However, to ensure that the DVArray instance doesn't get
destroyed while the NumPy array still exists, I need to increase the
reference count of the DVArray-wrapped Python instance.

 > I am not familiar with the internals of NumPy (and I can't seem to 
 > get to the manual over the web), so I can't tell you whether this is 
 > possible. The NumPy array would need to have some slot where you 
 > could store a pointer to the DVArray (if you only store the &self[0] 
 > you can't get back to the DVArray), and it would also need to give 
 > you a way to hook its destruction.

It has such a slot, base (a PyObject*), whose reference count gets decremented when
the NumPy array is deleted, which is exactly what I want.

 > A DVArray object is-a PyObject, so you can always write:
 > 
 > PyObject* to_array(DVArray& self) {
 >    PyObject* p = &self; // get the PyObject*
 >    Py_INCREF(p); // increment, decrement, whatever

This is exactly what I needed to know. I was confused about the usage
of DVArray in the to_array function. I thought DVArray referred to
the the valarray<double> class and not the wrapped valarray<double>
class. If I wanted a pointer to a valarray<double> (i.e., DVArray) and
not the wrapped valarray<double> class, how would I get it in this
function? In general, I'm confused about when a class is considered
wrapped and when it's not in the context of functions added as methods
of the wrapped class.

 > In any case, Python usually doesn't care about the actual types of 
 > objects (just their interfaces), so you might find it much easier to 
 > build up the interface of DVArray so that it mimics a NumPy array and 
 > can be used in its place.

NumPy arrays have a rich interface and allows things like sin(x),
where x is an NumPy array, to be computed very quickly. To implement
that interface and functionality for valarray<> or vector<> would be a
large amount of work, hence, my use of the above strategy.

Thanks very much for your help and for BPL -- it's a fantastic piece
of work.

John

-- 
John Barnard
Assistant Professor
Department of Statistics
Harvard University
Phone: [phone]
Fax:   [phone]
Email: barnard_at_[hidden]