[Boost Python Library] Mixing types wrapped in different modules

From: rwgk_at_[hidden]
Date: 2000-12-07 10:02:09

This message is about an informal proposal for adding a provision to
the Boost Python Library (BPL) that would allow mixing types (i.e.
extension classes) defined in different (separately imported) Python
modules.

Attached is a copy of a conversation with David Abrahams. On request
of David, I continue the conversation in this forum.

From the conversation below:
  David:
  : I think it's overkill. Do you really need to put these things in 
separate
  : modules?

We will be using the Boost Python Library in a fairly major software
project in the field of x-ray crystallography. There will be at least
four independent research groups working on this project. Some groups
do not want to share all their source code.

I anticipate that the final package will contain on the order of 50-
100
objects for which we want Python bindings via BPL. Each research group
will contribute a certain number of objects. There will be very 
complex
objects, and complex relationships between the objects. It is not
unlikely that a high level extension module will rely on, say, 5-10
other modules.

Without a provision similar to the import_class_builder proposed 
below,
one large BPL module that wraps all our objects would have to be
maintained. This module would constantly need to be updated by a 
number
of developers, and all precompiled objects would need to be in place
for linking. Sharing a new high-level object with Python bindings
between research groups that will inevitably have diverging versions 
of
the master BPL module would be very painful. Our entire system would
not be modular. We would lose most of the benefits of dynamic linking.

In contrast, sharing modules that establish relationships at run time
would lead to a modular and much more maintainable system. We could
simply mail the .so or .dll files with the latest fixes and in many
cases our collaborators might be able to use them without recompiling.
At a later stage, supplying bug fixes for production systems would be
greatly simplified because only the defective .so or .dll files would
need to be replaced. This is important because there could be a 
variety
of production systems based on the same basic building blocks, but 
with
different higher level capabilities. Another advantage of a more
modular system design is that it will be easier for "casual 
developers"
to contribute new functionality in form of a new module. In our branch
of science, the casual developer is the norm, and full time computer
scientists are the exception. So again, this is an important argument
for us.

Comments are appreciated!

Ralf

======================================================================
========
======================================================================
========

Due to the problems with the CVS access, the *getstart6* files
mentioned below are not yet easily accessible to everybody, but I 
think
this is not very important for understanding the subject of the
discussion.

======================================================================
========
======================================================================
========
Ralf:
> The *getstart6* files in the build/tru64 directory are the 
intermediate
> result of a crude first attempt to mix types defined in different
> modules. I.e., getstart6a is an extension module that exposes ivect,
> and getstart6b is an extension module that exposes dvect.  Now I
> would like to provide methods ivect.as_dvect() and dvect.as_ivect(),
> but for maintainability (in a large project) I do not want to merge
> getstart6?.cpp into one file.

David:
: You could still put them in the same module without having a single 
source
: file. That would make things much easier on you. You just need to 
be sure
: that class_builder<dvect> and class_builder<ivect> are instantiated 
in both
: source files. If you have a source file that needs from_/to_python()
: functions for wrapped classes Foo1,... FooN, but where you don't 
want to
: actually do the wrapping for Foo1,...FooN, you could do something 
like this:
: 
: struct FooInstantiator : boost::python::class_builder<Foo1>,
: boost::python::class_builder<Foo2>,...
:                                      
boost::python::class_builder<FooN>
: {
:     FooInstantator();
: }

Ralf:
> After thinking about this for a while I came to the conclusion that
> there needs to be another version of class_builder<class T>, e.g.
> something like "import_class_builder<class T, char *module>".
> dvect_class in getstart6a would be an instance of 
import_class_builder.
> This import_class_builder would need to import the other module
> that wraps T, in this case the module getstart6b, and somehow get
> a handle on the wrapped T, in this case getstart6b.dvect. The second
> thing import_class_builder has to do is to define a python::type<> 
for
> dvect in a similar way the regular class_builder does.
>
> My particular getstart6 example is complicated by the fact that
> getstart6a would import getstart6b and vice versa. Being able
> to handle this kind of cyclic dependency would be nice, but does
> not seem essential to me. At least not initially.
>
> Does my idea of an import_class_builder make sense?

David:
: I think it's overkill. Do you really need to put these things in 
separate
: modules?

Ralf:
> Do you have any thoughts on, or plans for implementing this?

David:
: Not yet ;-)
======================================================================
========
======================================================================
========