Re: [boost] how to destroy a *real* copy of a python object from c++ (or maybe i want something else ?)

From: David Abrahams (david.abrahams_at_[hidden])
Date: 2001-11-27 15:04:08

----- Original Message -----
From: "daniel kottow" <daniel.kottow_at_[hidden]>

> hi dave,
> sorry i am back, but my hill-climbing seems to have failed before the real
> hill even started ...
> so this time i will start by giving you the big picture:
> the genetic-algorithm library i want to python-enable is based on two
> objects:
> lets call them Genome and Algorithm.

> Algorithm maintains a pool of Genomes, calls their objective function and
> does the genetic operations like selection, cross-over and such. a typical
> (pseudo)coding example would be
>
> float Objective(const Genome &g)
>   file://return something fancy based on props of g
>
> Genome g(init_props, Objective);
> Algorithm a(g, init_props);
> for(;;)
>   a.step();
>
> AYCC, Algorithm uses an example genome for building the population.
> for this, it relies on virtual member functions called clone and copy.
> signatures:
>
> virtual Genome *Genome::clone() const;
> virtual void Genome::copy(const Genome &g)
>
> which are suppposed to be overriden if you derive your own Genome. it
first
> clones your example genome to get an initial population and then mainly
> uses copy while applying genetic ops.
> i thought it would be a nice idea to make Objective() a virtual member
> function in a derived class and then expose it to python.
>
> so i did:
>
> void objective(const Genome &g)
> {
> ((PyGenome&)g).objective();
> }

I suggest static_cast<> or dynamic_cast<> here, though I wonder why not just
have an objective() member function in Genome itself.

> class PyGenome : public Genome {
> Genome *clone();
> // void copy(const Genome &g); file://no need to change anything here,
> use base class imp
> virtual float objective();
> };
>
> and implemented (trying to use your suggestions):
>
> Genome *Genome::clone() {
>    python::ref *pr;
>    pr = new python::ref(newPythonUserGenome());
>    return
> BOOST_PYTHON_CONVERSION::from_python(pr->get(),
>   python::type<PyGenome*>());
> }

Well, OK, but now you're just leaking that dynamically-allocated ref (*pr),
which holds a reference to the result of newPythonUserGenome(). So, you
might as well just call Py_INCREF() on the result of newPythonUserGenome()
and leak that without creating a ref.

> and
>
> python::ref PyGenome::newPythonUserGenome() const {
>
>   return python::callback<python::ref>::call(pyFuncNewGenome.get());
> }
>
> where pyFuncNewGenome is a python::ref which must point to the python
> constructor (without args) of the concrete python class which implements
> the virtual func objective. (see last posting)

Actually, if the __init__ function is what you mean by "the constructor",
pyFuncNewGenome points to the class itself. You can't call the __init__
function without an instance, but the class constructs and initializes
instances when called.

> i think the rest of the boost::python code (callback stuff and so on) is
> pretty straightforward,
> as is the python-enabling of Algorithm.
> (but if you need to see more, i can tell/send you more, ofcourse)
>
> runtime resume:
>
> the algorithm basically works ok.
> BUT: if an object returned by clone is destroyed by C++ i immediately get
a
> crash.
> i understand this behaviour, but i dont know how to change it.
>
> actually, first i did not create a python::ref in heap and this would
crash
> much earlier
> (because the python::ref would go out of scope and destroy the object,
> right? )
>
> now its the other way around, python thinks the object lives forever but
it
> doesnt.
>
> now, i hope you can give me some advice on this, may it either be
regarding
> the design or technical.
> i was very much reminded by this experience how both things should not go
> apart,
> specially if the environment is new to you (thats me).

There's only one way I can think of to deal with this. The issue is that
when you have a C++ object and a corresponding Python object, only one of
them can be in control (unless you have true GC, which we don't). In
Boost.Python the master is the Python object. It looks to me as though you
want to reverse the relationship.

So, I propose that you write a new class derived from Genome which holds a
boost::python::ref to the Python object, and which implements the Genome
interface by forwarding all functions to the Genome object held by the
Python object it holds.

Thus:

    Genome - an abstract base class
    GenomeFwd : public Genome - holds a Genome* g, forwards interface to *g
    GenomeFwdToPy : public GenomeFwd - holds a boost::python::ref
        constructor gets the Genome* out of the ref and constructs base
class

    GenomeCallback : public Genome - used to wrap Genome for Python derivers

HTH,
Dave