Re: [Boost-users] Exposing C++ Instances to Python

Subject: Re: [Boost-users] Exposing C++ Instances to Python
From: Leonhard Weber (lweberk_at_[hidden])
Date: 2010-05-28 12:53:28

Hi again,

Thank you for your precious help up to now. I figured I had to see more and
take more time working things out by myself before actually making use of
your precious time.
I've spent the last days building a testing environment to simulate the
requirements of the Interface Implementation that is waiting for me.

The Idea:
An Interface that should make communication between the SpringRTS Engine and
Python AI developers possible without tainting the SpringRTS sourcetree, and
encapsulating as much as possible without invading or dictating "how to do
it" on the Python side.
1) Engine starts the Interface (C/C++)
2) Engine tells the Interface to start a AI, passes a Callback that the
Interface has to make accessible for the Python side
3) Interface starts the Python Interpreter and starts the actual AI
4) From Python, through the __init__.py the Client of the Interface fetches
for the Callback instance and also registers some functions that are called
from the Engine on ongoing happening Events (The topic that comes after
finishing the current one which is encapsulating/automating the SAIFloat3 to
and from PyTuple's)
5) The rest is up to the Client and the Engine

For now the Interface is initialized from python.

Here is the Code: (The entire thing is a OpenSource Project, so I don't
really mind on publicly making this an example for other inexperienced users
as myself)
################################## C++ Start
#include <stdlib.h>
#include <boost/python.hpp>
#include <boost/python/call.hpp>
#include <boost/python/ptr.hpp>
#include <boost/python/register_ptr_to_python.hpp>

using namespace boost;
using namespace boost::python;

// =============================================================
//     Defining the to be passed Callback and SAIFloat3.
//         Callback:
//             Testing reference passing etc...
//
//         SAIFloat3:
//             Testing of a nested custom Type (How to Wrap)
//
//      Also some automatic converters for SAIFloat3 <--> Tuple
// =============================================================
struct SAIFloat3 // As defined in SAIFloat3.h (SpringRTS ExternalAI
Interface)
{
    float x, y, z;
};

// DUMMY CALLBACK FOR TESTING PURPOSES
struct EngineCallback
{
    void setInt(int number) { this->number = number; }
    void setLoop(bool loop) { this->loop = loop; }
    int getInt() const { return this->number; }
    bool getLoop() const { return this->loop; }

    void setVec1(float x, float y, float z) { vec1.x = x; vec1.y = y; vec1.z
= z; }
    void setVec2(float x, float y, float z) { vec2.x = x; vec2.y = y; vec2.z
= z; }
    SAIFloat3 getVec1() const { return vec1; }
    SAIFloat3 getVec2() const { return vec2; }

    SAIFloat3 vec1;
    SAIFloat3 vec2;
    int number;
    bool loop;
}; // EngineCallback

struct float3_to_python_tuple
{
    static PyObject* convert(SAIFloat3 const& float3)
    {
        boost::python::list result;
        result.append(object(float3.x));
        result.append(object(float3.y));
        result.append(object(float3.z));

        return boost::python::incref(boost::python::tuple(result).ptr());
    }

    static const PyTypeObject* get_pytype() { return &PyTuple_Type; }
};

struct float3_from_python_tuple
{
    float3_from_python_tuple()
    {
        boost::python::converter::registry::push_back(&convertible,
                                                      &construct,

boost::python::type_id<SAIFloat3>());
    }

    static void* convertible(PyObject* obj_ptr)
    {
        if (!(PyTuple_Check(obj_ptr)))
            return 0;

        int obj_size = PyObject_Length(obj_ptr);
        if (!(obj_size == 3) ) { // must contain 3 elements
            PyErr_Clear();
            return 0;
        }

        return obj_ptr;
    }

    static void construct(PyObject* obj_ptr,

boost::python::converter::rvalue_from_python_stage1_data* data)
    {
        void* storage = (

(boost::python::converter::rvalue_from_python_storage<SAIFloat3>*)
            data)->storage.bytes;
        new (storage) SAIFloat3();
        data->convertible = storage;
        SAIFloat3& result = *((SAIFloat3*)storage);

        result.x = extract<float>(PyTuple_GetItem(obj_ptr, 0));
        result.y = extract<float>(PyTuple_GetItem(obj_ptr, 1));
        result.z = extract<float>(PyTuple_GetItem(obj_ptr, 2));

        if (PyErr_Occurred())
            boost::python::throw_error_already_set();
    }
};

// =============================================================
//     Initializator and Instanceholder of a EngineCallback
//  object.
// =============================================================
class PythonInterface {
    public:
        PythonInterface();

        EngineCallback* exportCallback();

    private:
        EngineCallback* instance_callback;
};

PythonInterface::PythonInterface() {
    this->instance_callback = new EngineCallback;
    this->instance_callback->setInt(5);
    this->instance_callback->setLoop(true);
}

EngineCallback* PythonInterface::exportCallback() {
    return this->instance_callback;
}

//  ===================
//     Wrapping all up
//  ===================
BOOST_PYTHON_MODULE(PythonInterface)
{
    to_python_converter< SAIFloat3, float3_to_python_tuple
#ifdef BOOST_PYTHON_SUPPORTS_PY_SIGNATURES
        , true
#endif
        >();
    float3_from_python_tuple();

    class_< SAIFloat3 >( "SAIFloat3" )
        .def_readwrite("x", &SAIFloat3::x)
        .def_readwrite("y", &SAIFloat3::y)
        .def_readwrite("z", &SAIFloat3::z)
        ;

    class_< PythonInterface, shared_ptr<PythonInterface>
>("PythonInterface")
        .def("exportCallback", &PythonInterface::exportCallback,
return_value_policy<reference_existing_object>())
    ;

    class_< EngineCallback >("EngineCallback")
        .def("setInt", &EngineCallback::setInt)
        .def("setLoop", &EngineCallback::setLoop)
        .def("getInt", &EngineCallback::getInt)
        .def("getLoop", &EngineCallback::getLoop)

        // RESPONSIBLE FOR THE COMPILE ERROR
        .def("setVec1", &EngineCallback::setVec1)
        .def("setVec2", &EngineCallback::setVec2)
        .def("getVec1", &EngineCallback::getVec1,
return_value_policy<return_by_value>())
        .def("getVec2", &EngineCallback::getVec2,
return_value_policy<return_by_value>())
//         .def("getVec1", &EngineCallback::getVec1,
return_value_policy<copy_const_reference>())
//         .def("getVec2", &EngineCallback::getVec2,
return_value_policy<copy_const_reference>())
//         .def("getVec1", &EngineCallback::getVec1,
return_value_policy<reference_existing_object>())
//         .def("getVec2", &EngineCallback::getVec2,
return_value_policy<reference_existing_object>())
    ;
}
// copy_const_reference
################################## C++ End

Python testscript:
################################## Python Start
from PythonInterface import PythonInterface
>>> __main__:2: RuntimeWarning: to-Python converter for SAIFloat3 already
registered; second conversion method ignored.

interface = PythonInterface()
callback = interface.exportCallback()

vec1 = callback.getVec1()

>>> vec1
<PythonInterface.SAIFloat3 object at 0x7f5959544410>
>>> type(vec1)
<class 'PythonInterface.SAIFloat3'> # Not exactly what I have in mind
################################## Python End

With that Error I don't even get close to testing the actual conversion code
which is an adapted version of
scitbx/boost_python/container_conversions.h. The main modification being,
cutting on the generics and the endless Python
Types checking (reduced to 'is it a tuple?')
To be honest am not experienced enough in C/C++ to see if it works without
actually feeling the pain.

Another thing is the different return_value_policies: by_value,
copy_const_reference(recommendet in the documentation for its efficiency),
reference_existing_object (which gives a reference to the actual object
omitting the desired conversion).
On <copy_const_reference> it shrieks at compiletime:

################################## Error Start
In file included from
/usr/include/boost/preprocessor/iteration/detail/iter/forward1.hpp:47,
                 from /usr/include/boost/python/detail/invoke.hpp:63,
                 from /usr/include/boost/python/detail/caller.hpp:16,
                 from
/usr/include/boost/python/object/function_handle.hpp:8,
                 from
/usr/include/boost/python/converter/arg_to_python.hpp:19,
                 from /usr/include/boost/python/call.hpp:15,
                 from /usr/include/boost/python/object_core.hpp:12,
                 from /usr/include/boost/python/args.hpp:25,
                 from /usr/include/boost/python.hpp:11,
                 from PythonInterface.cpp:2:
/usr/include/boost/python/detail/invoke.hpp: In function ‘PyObject*
boost::python::detail::invoke(boost::python::detail::invoke_tag_<false,
true>, const RC&, F&, TC&) [with RC =
boost::python::detail::caller_arity<1u>::impl<F, Policies,
Sig>::operator()(PyObject*, PyObject*) [with F = SAIFloat3
(EngineCallback::*)()const, Policies =
boost::python::return_value_policy<boost::python::copy_const_reference,
boost::python::default_call_policies>, Sig = boost::mpl::vector2<SAIFloat3,
EngineCallback&>]::result_converter, F = SAIFloat3
(EngineCallback::*)()const, TC =
boost::python::detail::caller_arity<1u>::impl<F, Policies,
Sig>::operator()(PyObject*, PyObject*) [with F = SAIFloat3
(EngineCallback::*)()const, Policies =
boost::python::return_value_policy<boost::python::copy_const_reference,
boost::python::default_call_policies>, Sig = boost::mpl::vector2<SAIFloat3,
EngineCallback&>]::c_t0]’:
/usr/include/boost/python/detail/caller.hpp:223:   instantiated from
‘PyObject* boost::python::detail::caller_arity<1u>::impl<F, Policies,
Sig>::operator()(PyObject*, PyObject*) [with F = SAIFloat3
(EngineCallback::*)()const, Policies =
boost::python::return_value_policy<boost::python::copy_const_reference,
boost::python::default_call_policies>, Sig = boost::mpl::vector2<SAIFloat3,
EngineCallback&>]’
/usr/include/boost/python/object/py_function.hpp:38:   instantiated from
‘PyObject*
boost::python::objects::caller_py_function_impl<Caller>::operator()(PyObject*,
PyObject*) [with Caller = boost::python::detail::caller<SAIFloat3
(EngineCallback::*)()const,
boost::python::return_value_policy<boost::python::copy_const_reference,
boost::python::default_call_policies>, boost::mpl::vector2<SAIFloat3,
EngineCallback&> >]’
PythonInterface.cpp:167:   instantiated from here
/usr/include/boost/python/detail/invoke.hpp:88: error: no match for call to
‘(const
boost::python::detail::copy_const_reference_expects_a_const_reference_return_type<SAIFloat3>)
(SAIFloat3)’
In file included from
/usr/include/boost/python/object/function_handle.hpp:8,
                 from
/usr/include/boost/python/converter/arg_to_python.hpp:19,
                 from /usr/include/boost/python/call.hpp:15,
                 from /usr/include/boost/python/object_core.hpp:12,
                 from /usr/include/boost/python/args.hpp:25,
                 from /usr/include/boost/python.hpp:11,
                 from PythonInterface.cpp:2:
/usr/include/boost/python/detail/caller.hpp: In static member function
‘static const PyTypeObject*
boost::python::detail::converter_target_type<ResultConverter>::get_pytype()
[with ResultConverter =
boost::python::detail::copy_const_reference_expects_a_const_reference_return_type<SAIFloat3>]’:
/usr/include/boost/python/detail/caller.hpp:242:   instantiated from ‘static
boost::python::detail::py_func_sig_info
boost::python::detail::caller_arity<1u>::impl<F, Policies, Sig>::signature()
[with F = SAIFloat3 (EngineCallback::*)()const, Policies =
boost::python::return_value_policy<boost::python::copy_const_reference,
boost::python::default_call_policies>, Sig = boost::mpl::vector2<SAIFloat3,
EngineCallback&>]’
/usr/include/boost/python/object/py_function.hpp:48:   instantiated from
‘boost::python::detail::py_func_sig_info
boost::python::objects::caller_py_function_impl<Caller>::signature() const
[with Caller = boost::python::detail::caller<SAIFloat3
(EngineCallback::*)()const,
boost::python::return_value_policy<boost::python::copy_const_reference,
boost::python::default_call_policies>, boost::mpl::vector2<SAIFloat3,
EngineCallback&> >]’
PythonInterface.cpp:167:   instantiated from here
/usr/include/boost/python/detail/caller.hpp:102: error: ‘struct
boost::python::detail::caller_arity<1u>::impl<F, Policies,
Sig>::operator()(PyObject*, PyObject*) [with F = SAIFloat3
(EngineCallback::*)()const, Policies =
boost::python::return_value_policy<boost::python::copy_const_reference,
boost::python::default_call_policies>, Sig = boost::mpl::vector2<SAIFloat3,
EngineCallback&>]::result_converter’ has no member named ‘get_pytype’
################################## Error End

Where are the mistakes? Where did I screw again? What concepts am I not
understanding?

Thanks,
~lwk

On 25 May 2010 18:16, XiongJia Le <lexiongjia_at_[hidden]> wrote:

> Are you finding  "Boost.Python.register_ptr_to_python" ?
> Please look this document
> http://www.boost.org/doc/libs/1_41_0/libs/python/doc/v2/register_ptr_to_python.html
>
> Is this sample same as you require ?
>
> Thanks,
> Xiongjia Le ( http://notes.xj-labs.net/ )
>
>
> On Tue, May 25, 2010 at 11:34 PM, Leonhard Weber <lweberk_at_[hidden]>wrote:
>
>> Made a mistake... srry for my lacking knowledge/experience in C++. Trying
>> to improve :D.
>>
>> For test purposes I changed the Python Script so it inits an instance, and
>> then tries to fetch the instance through the PyCallback:
>> #####
>> from EngineCallback import *
>>
>> a = EngineCallback()
>> b = PyCallback(a)
>>
>> >>> a
>> <EngineCallback.EngineCallback object at 0x7ffbe74c5208>
>> >>> b
>> <EngineCallback.EngineCallback object at 0x7ffbe74b0050>
>>
>> #####
>> On the C++ side I had to change this, its now passing the argument by
>> value (which I dont want)
>> shared_ptr<EngineCallback> PyCallback( EngineCallback* test ) { return
>> shared_ptr<EngineCallback>(test); }
>>
>> If I passed it per reference EngineCallback* &test (hope Im getting the
>> jargon right and not mixing up definitions) it would say:
>>
>>
>> did not match C++ signature:
>>     PyCallback(EngineCallback* {lvalue})
>>
>> by changing it to pass by value it works, though isnt it making a copy of
>> the instance? how do I get just a reference to the instance passed to
>> Python?
>>
>>
>>
>>
>> On 25 May 2010 16:56, Steven Watanabe <watanabesj_at_[hidden]> wrote:
>>
>>> AMDG
>>>
>>> Leonhard Weber wrote:
>>>
>>>> EngineCallback* test = new EngineCallback;
>>>>
>>>> // Entrypoint for Python to come ant fetch the Application Callback
>>>> shared_ptr<EngineCallback> PyCallback( EngineCallback* &test );
>>>> <snip>
>>>>
>>>> # test.py
>>>> from EngineCallback import PyCallback
>>>> from EngineCallback import EngineCallback
>>>>
>>>> test = PyCallback()
>>>>
>>>>
>>>
>>> PyCallback takes one argument.  Hence the exception below.
>>> Did you intend to use the global "test" instead of taking test
>>> as an argument in PyCallback?
>>>
>>>
>>>  Traceback (most recent call last):
>>>>  File "test.py", line 5, in <module>
>>>>    test = PyCallback()
>>>> Boost.Python.ArgumentError: Python argument types in
>>>>    EngineCallback.PyCallback()
>>>> did not match C++ signature:
>>>>    PyCallback(EngineCallback* {lvalue})
>>>>
>>>>
>>>
>>> In Christ,
>>> Steven Watanabe
>>>
>>> _______________________________________________
>>> Boost-users mailing list
>>> Boost-users_at_[hidden]
>>> http://lists.boost.org/mailman/listinfo.cgi/boost-users
>>>
>>
>>
>> _______________________________________________
>> Boost-users mailing list
>> Boost-users_at_[hidden]
>> http://lists.boost.org/mailman/listinfo.cgi/boost-users
>>
>
>
> _______________________________________________
> Boost-users mailing list
> Boost-users_at_[hidden]
> http://lists.boost.org/mailman/listinfo.cgi/boost-users
>