Re: [boost] how to get a *real* copy of a boost-python object that supports virtual function

From: David Abrahams (david.abrahams_at_[hidden])
Date: 2001-11-24 01:00:07

----- Original Message -----
From: <dkottowk_at_[hidden]>
To: <boost_at_[hidden]>
Sent: Friday, November 23, 2001 3:18 PM
Subject: [boost] how to get a *real* copy of a boost-python object that
supports virtual function

> hi all.
> i am having difficulties with handling the copy of python-boosted objects
FROM c++:

Do you mean that you want to generate a new wrapped copy of a wrapped C++
object? That's normally done by simply returning the copied object (or a
const& to the original object) from a function.

> consider the following definitons, belonging to a genetic algorithm (GA)
library:
>
> class PyGenome : public Genome {
>
> public:
>
>    PyGenome(int width);
>
>   PyGenome( const PyGenome & data);
>
>   virtual float objective();
>
>   virtual void copy(const Genome &data);
>   virtual Genome * clone(Genome::CloneMethod = CONTENTS) const;
>
>   PyObject *genes();
>
> };
>
> /* we need a derived class to handle virtual funcs with boost... */
> class PyGenome_Callback : public PyGenome {
>
> public:
>   PyGenome_Callback(PyObject* self_, int width);
>
>   PyGenome_Callback(PyObject *self_, const PyGenome & data);
>
>   virtual void copy(const Genome &data);
>   virtual Genome * clone(Genome::CloneMethod = CONTENTS) const;
>
>   float objective();
>   static float default_objective( PyGenome& self_);
>
> private:
>   PyObject *self;
> };
>
> the goal here is to be able to override the objective function objective()
in python.
> by that way we get a general purpose GA for use in python.

It looks like you've done everything right so far...

> it is in the nature of GA that it has to copy/clone the genomes.
> this occurrs from inside the algorithm, that is, from c++ code.
> i did follow the tutorial, step "Overridable Virtual Functions" when
coding.
>
> i provided copy/clone for PyGenome and defined the same in
PyGenome_Callback as follows:
>
> PyGenome_Callback::PyGenome_Callback(PyObject *self_, const PyGenome &
data)
>   : PyGenome(data), self(self_)  {}
>
> void pyGenome_Callback::copy(const Genome &data) {
>   cout << "called ga_callback copy " << endl;
>   PyGenome::copy(data);
>   cout << "self " << self << " this " << this << endl;
>   self = ((const PyGenome_Callback &)data).self; // unnecessary ??

First of all, I don't know if that's legal. Are you sure that you're always
dealing with a wrapped PyGenome here?
Secondly, not only is it unneccessary, it's probably incorrect. The self
member is storing a pointer back to the PyObject which holds the PyGenome
object. What you've now got is something like this:

+------------+      +---------------------+
| PyObject A +----->| PyGenome_Callback A |
|            |   +--|                     |
+------------+   |  +---------------------+
           +-----+
+----------V-+      +---------------------+
| PyObject B +----->| PyGenome_Callback B |
|            |<-----|                     |
+------------+      +---------------------+

> }
>
> GAGenome *  PyGenome_Callback::clone(GAGenome::CloneMethod) const {
>   cout << "called ga_callback clone " << endl;
>   cout << "self " << self << " this " << this << endl;
>   return new PyGenome_Callback(self, *this);
> }
>

I think that making new dynamically-allocated instances of PyGenome_Callback
is probably also misguided. The xxx_Callback classes in the formula for
making overridable virtual functions are only really there for the benefit
of Boost.Python, so that virtual functions can be hooked into the Python
override mechanism. Assuming PyGenome_Callback is only supposed to be used
as a "base class" for Python subclasses of PyGenome, then you probably want
to create a new Python object of the same class as the leaf being cloned
from the Python hierarchy.

In the world of Boost.Python, it's the Python object which gets control over
the lifetime of its C++ base class component(s), so you'd need to do
something very different to implement cloning semantics. I'm not sure
exactly what you should do, in part because I can't see the rest of what
you're working on, but you would need clone() to return some kind of smart
pointer which manages a Python object (e.g. boost::python::ref), and use
from_python(p.get(), type<PyGenome&>()) or something to get the C++ object
out of it.

> float PyGenome_Callback::objective()

>   return boost::python::callback<float>::call_method(self, "objective");
> }
>
> now, the copy/clone methods from PyGenome_Callback are used by the GA when
copy/cloning as i could verify with the debug output.
> actually i get many objects and each one has its own data.
> but when the python objective fn is called, from inside the c++ code,
> i always see the same data.

That's probably because you're resetting the self pointer of all copied
objects to refer to the same Python object.

> i verified this by calling the exposed method genes().
> actually the debug gives always the same mem location for the python
reference self,
> although the this pointer obviously varies.
> my question is, how do i get a copy of PyGenome_Callback,
> which also is a copy of the python-side of the object ?

Most of what you're asking is still unclear to me, but I think you're not
considering what has to happen under the covers carefully enough yet. Please
have a look at http://www.boost.org/libs/python/doc/data_structures.txt to
see if that helps clear things up a bit.

> greetings, daniel.
>
> PS: i read in your tutorial also that non-intrusiveness is *the* major
goal for boost-python.

The tutorial doesn't say that. It's *a* major goal. Boost.Python can be used
other ways.

> i believe people (as me) also consider using boost-python for developing
directly extensions for python.

Yes, many do.

> actually, as i found out when trying to port this GA library, it might not
make so much sense to have as many types in python as you usually get in
c++.

I'm not sure what you mean by that. Could you explain?

> and you would also want to take advantage of python built-in types like
lists, so hand-crafting makes a lot of sense to me, anyway.
>

What, specifically, are you suggesting?

You can use boost::python::list, boost::python::string,
boost::python::tuple, boost::python::dict, etc. arguments to your C++
functions and wrap them as usual, if that helps...

-Dave