[Boost-users] boost::python -- passing containers of shared pointers as arguments and getting automatic conversion

Subject: [Boost-users] boost::python -- passing containers of shared pointers as arguments and getting automatic conversion
From: Eric Jonas (jonas_at_[hidden])
Date: 2008-10-07 20:19:47

Hello! 

I've found several examples of code on the internet where people are
trying to convert from a native python container type to a
std::container. Using those as a template, I'm trying to convert from a 
python tuple of objects wrapped by boost::python to a std::vector of 
boost::shared_ptrs. 

Example c++: 
http://pastebin.com/m3b198a9c

Example python showing the code running: 

http://pastebin.com/m400f17e1

Example output:
http://pastebin.com/m1b0eb94e

[all pasted at the end of the mail]

[Please note these only work if you've hacked your boost/shared_ptr.hpp
to make everything public]

Unfortunately, I'm running into a problem with extract<shared_ptr<T>>
creating a -new- shared pointer pointing to the raw pointer -within- the
existing shared pointer -- so no reference-comanaging occurs. I've
hacked my shared_ptr.hpp to let me examine the refcounted struct inside
the shared pointer. 

What I see when I run things is as follows: 

[jonas_at_hamiltonian core]$ python broken_test.py 
Foo 0 is at 88276256 with reference counter 8276288
Foo 1 is at 88276352 with reference counter 8276384
Foo 2 is at 88276320 with reference counter 8276416
Extraction of object pointing to 88276256 but with reference counter
0x7f54c0
Extraction of object pointing to 88276352 but with reference counter
0x7f3f80
Extraction of object pointing to 88276320 but with reference counter
0x7e4a00

Note what this is saying: While internal to my FooHolder, the
shared pointers point to location X and reference-counting struct Y, 
within the converter they're being turned into new shared pointers,
still managing the raw pointer X but with a different
reference-counting-struct Z.

This is most perplexing, and I worry it might be the result of
interaction with the fact that in my example code, I'm explicitly using
the shared pointer to manage the reference count: 
 
  class_<Foo, pFoo_t>("Foo"); 

In any case, I'm quite stumped, and have a -ton- of places in my main
codebase where stl containers are passed around containing shared_ptrs. 
I asked around on IRC but everyone seemed stumped too. Any help would be
greatly appreciated. 

Thanks!
                ...Eric Jonas

broken.cc: 

        #include <boost/python.hpp>
        #include <boost/python/class.hpp>
        #include <boost/python/converter/shared_ptr_from_python.hpp>
        #include <boost/python/implicit.hpp>
        #include <boost/python/module.hpp>
        #include <boost/variant/apply_visitor.hpp>
        #include <boost/shared_ptr.hpp>
        #include <vector>
         
        using namespace boost::python;
        using namespace std; 
         
        template <class T1>
        struct vector_from_tuple_of_shared
        {
          vector_from_tuple_of_shared()
          {
            converter::registry::push_back(&convertible, &construct,  
        
         boost::python::type_id<std::vector<T1> >());
          }
          
          static void* convertible(PyObject* obj_ptr)
          {
            handle<> x(borrowed(obj_ptr));
            boost::python::tuple o(x);
            for (int i = 0; i < boost::python::len(o); i++) {
              
              extract<T1> obj(o[i]); 
              if (!obj.check() ) {
                return 0; 
              }
         
            }
            return obj_ptr;
          }
          
          static void construct(PyObject* obj_ptr,
        
        converter::rvalue_from_python_stage1_data* data)
          {          
            handle<> x(borrowed(obj_ptr));
            std::vector<T1> value; 
         
            boost::python::tuple o(x);
            for (int i = 0; i < boost::python::len(o); i++) {
              
              extract<T1> obj(o[i]); 
              T1 s1 = obj; 
              std::cout << "Extraction of object pointing to " <<
        ios::hex
                        << (int64_t) s1.get() << " but with reference
        counter "
                        << s1.pn.pi_ << std::endl; 
              value.push_back(obj); 
         
            }
         
            void* storage =
        ( (converter::rvalue_from_python_storage<std::vector<T1> >*)
        data)->storage.bytes;
            new (storage) std::vector<T1>(value);
            data->convertible = storage;
            
          }
        };
         
         
        class Foo
        {
          
         
        }; 
         
        typedef boost::shared_ptr<Foo> pFoo_t; 
         
        class FooHolder
        {
        public:
          std::vector<pFoo_t> foos_; 
          pFoo_t createFoo() {
            pFoo_t f(new Foo()); 
            foos_.push_back(f); 
            return f; 
         
          }
         
          void printFooStatus() {
            for(int i = 0; i < foos_.size(); i++) {
              std::cout << "Foo " << i << " is at " << ios::hex
                        << (int64_t)foos_[i].get() << " with reference
        counter " 
                        << (int64_t)foos_[i].pn.pi_ << std::endl; 
            }
          }
          
        };
         
         
        void doStuff(std::vector<pFoo_t> stuff)
        {
          
         
        }
         
        BOOST_PYTHON_MODULE(broken)
        {
          class_<Foo, pFoo_t>("Foo"); 
          class_<FooHolder>("FooHolder")
            .def("createFoo", &FooHolder::createFoo)
            .def("printFooStatus",  &FooHolder::printFooStatus)
            ;
          
          
          def("doStuff", &doStuff); 
         
          vector_from_tuple_of_shared<pFoo_t>(); 
         
          
        }

Broken.py:

        import broken
         
        fh = broken.FooHolder()
         
        f1 = fh.createFoo()
        f2 = fh.createFoo()
        f3 = fh.createFoo()
         
        fh.printFooStatus()
         
        broken.doStuff((f1, f2, f3))

Output: 
[again, requires modified shared_ptr.hpp]

        jonas_at_hamiltonian core]$ make broken.so
        g++ -O2 -g -pg -I/usr/local/include -fPIC
        -I../threadpool/threadpool/include -pthread -fPIC
        -I/usr/include/`/usr/bin/python -c "import sys; (a, b, c, t, n)
        = sys.version_info; print 'python%d.%d' % (a, b),"`  -c
        broken.cc
        g++ broken.o -L/usr/local/lib  -lgsl -lgslcblas -lm
        -lboost_unit_test_framework -lboost_program_options -g -pg
        -lboost_thread  -lboost_signals -o broken.so -shared
        -I/usr/include/`/usr/bin/python -c "import sys; (a, b, c, t, n)
        = sys.version_info; print 'python%d.%d' % (a, b),"` \
                -L/usr/lib -lpython2.4 -lboost_python
         
         
        [jonas_at_hamiltonian core]$ python broken_test.py 
        Foo 0 is at 88276256 with reference counter 8276288
        Foo 1 is at 88276352 with reference counter 8276384
        Foo 2 is at 88276320 with reference counter 8276416
        Extraction of object pointing to 88276256 but with reference
        counter 0x7f54c0
        Extraction of object pointing to 88276352 but with reference
        counter 0x7f3f80
        Extraction of object pointing to 88276320 but with reference
        counter 0x7e4a00