Re: [Boost-build] Python Port Updates

Subject: Re: [Boost-build] Python Port Updates
From: Vladimir Prus (vladimir.prus_at_[hidden])
Date: 2016-09-27 13:19:22

Hi Aaron,

On 26-Sep-16 8:20 PM, aaron_at_[hidden] wrote:
> All,
>
> We have been using the Python port for over a year now and it seems to
> be working really well. It's super easy to make changes on the backend
> in Python and it's also easy to add functionality not possible before in
> Jam e.g. I/O calls, etc.

That is good to know!

> Having said that, the Python port is significantly slower than its Jam
> counterpart. I have been running the Python profiler against some of our
> larger projects and there are a number of things that stand out.

Ouch. Initial tests revealed that Python port is about 4x faster than 
Jam, but apparently Steven optimized Jam so well that the situation
is now reversed.

> The porting notes have highlighted some things that shouldn't be done
> e.g. "Don't use .has_key() on dictionaries, use the "in" keyword
> instead". There are a lot of other things that indicate that Python 2.2
> was probably the leading version of Python at the time. I would like to
> go through the codebase and update it all to try and follow most of the
> common Python idioms. Getters and Setters are used everywhere. In C++,
> getters and setters make sense, but in Python it adds a decent amount of
> overhead as there is the overhead of creating a new function frame on
> the stack. In my timeit tests, straight attribute access is about 4
> times faster than using a getter. Not to mention, if the attribute needs
> to have more calculations involved, Python provides data descriptors in
> order to turn the attribute access into a function call (usually with
> @property).

Fair enough.

> The biggest performance hit is the Features, Properties, and PropertySet
> instances. Based on the fact that there is a property_set module written
> in C, I'm assuming that Jam, at one point, suffered the same problem. I
> have a few ideas about how to solve this in pure Python, but it would
> require an extreme overhaul of the entire codebase as it would make the
> implementation of things different than that of Jam. If those ideas
> don't pan out, I will need to try to fall back to C like Jam does.

I think that yes, maybe some design change might be required to reduce
the number of PropertySet instances that are greated.

> Does anyone have a problem with the above issues?
> What will need to be done in order to accept the changes I make into the
> main repo?

You now have push access to the Boost.Build repository, and should be 
able to push changes directly, hopefully this will simplify things.
In addition, I'm likely to be working on Python port too during next 
month, so maybe that will be helpful too.

- Volodya