[Boost-build] Python Port Updates

Subject: [Boost-build] Python Port Updates
From: aaron_at_[hidden]
Date: 2016-09-26 13:20:48

All,
 
We have been using the Python port for over a year now and it seems to be working really well. It's super easy to make changes on the backend in Python and it's also easy to add functionality not possible before in Jam e.g. I/O calls, etc.
 
Having said that, the Python port is significantly slower than its Jam counterpart. I have been running the Python profiler against some of our larger projects and there are a number of things that stand out.
 
The porting notes have highlighted some things that shouldn't be done e.g. "Don't use .has_key() on dictionaries, use the "in" keyword instead". There are a lot of other things that indicate that Python 2.2 was probably the leading version of Python at the time. I would like to go through the codebase and update it all to try and follow most of the common Python idioms. Getters and Setters are used everywhere. In C++, getters and setters make sense, but in Python it adds a decent amount of overhead as there is the overhead of creating a new function frame on the stack. In my timeit tests, straight attribute access is about 4 times faster than using a getter. Not to mention, if the attribute needs to have more calculations involved, Python provides data descriptors in order to turn the attribute access into a function call (usually with @property).
 
The biggest performance hit is the Features, Properties, and PropertySet instances. Based on the fact that there is a property_set module written in C, I'm assuming that Jam, at one point, suffered the same problem. I have a few ideas about how to solve this in pure Python, but it would require an extreme overhaul of the entire codebase as it would make the implementation of things different than that of Jam. If those ideas don't pan out, I will need to try to fall back to C like Jam does.
 
There is a lot of power in the Python port and I would like to make it even more powerful by at least matching the speed of Jam. However, I don't like making changes to the core Boost.Build code without being able to push the changes back up stream as it removes maintenance of a fork on our end. So, I am asking for permission, thoughts, and guidance on taking on these issues.
 
Does anyone have a problem with the above issues?
What will need to be done in order to accept the changes I make into the main repo?
Other thoughts?
 
Thanks,
Aaron