Re: [jamboost] RFC: Proposal to port BBv2 to Python

From: Pedro Ferreira (pedro.ferreira_at_[hidden])
Date: 2005-01-21 05:34:39

Hi Volodya,

Em 20 Jan 2005, às 13:16, Vladimir Prus escreveu:

>> ----
>> Goal
>> ----
>> Create a Python-based Boost.Build library.
>> All the functionality available to BBv2 users shall be provided in 
>> the form
>> of a library with a well-documented API. Parsers for higher-level 
>> languages,
>> such as bjam, may be developed and use that API.
>> SCons shall be used as the build engine but its interface with the 
>> library
>> shall be abstracted out, avoiding mixing library code with direct 
>> calls to a
>> specific build engine, so that integration with other engines be 
>> possible
>> and simple.
>
> I think it's better to split "Boost.Build in Python" and "Boost.Build 
> in
> SCons" projects -- first convert to Python and only then to convert in 
> SCons.
> We can't just take SCons and use it. First of all, we must decide how 
> to best
> use it and if some of SCons design must be changed before we're able 
> to use
> it. We must also understand all the features it provides as build 
> engine to
> be sure we're using them all. And finally, I think that SCons needs 
> some
> performance work, too.

Right. But I hear that the SCons people are taking care of that, which 
is good news.

> So, switching to SCons is additional work. Maybe, we should first 
> switch to
> Python?

Agreed.

(...)

>
>> PyBB needs not, and should not, have the concept of a Jamfile or 
>> similar.
>> This would complicate its interface and add an additional burden for
>> applications that do not use "project" files.
>> However, the grouping of targets in projects makes sense because it 
>> is used
> in most real world applications. In addition to this, associating 
> names with
> projects make it easier to refer to other targets.
>>
>> --------- ------------------- ---------
>> | | 1 1 | | 1 * | |
>> | |<----------->| ProjectRegistry |----------->| Project |
>> | | | | | |
>> | | ------------------- ---------
>> | Manager | | 1
>> | | |
>> | | \|/ *
>> | | ---------
>> | | | |
>> --------- | Target |
>> | |
>> ---------
>
> One issue is that currently I plan to introduce per-project 
> id-to-project
> mapping (so "/boost" can mean different things in different projects). 
> Does
> this leave any functionality for the 'ProjectRegistry' class?

I'm not sure what you mean here.
Will this imply extended functionality for that class?
Will this render it unnecessary?

>> All of the above denote what we could call "static" configuration, 
>> that is,
>> it does not change in between several builds (provided that no more 
>> targets
>> are added, of course).
>> Before invoking the build system, PyBB needs to transform targets 
>> (PyBB
>> 'abstract' targets) in actual-targets (build system targets). In this
>> process, it also creates virtual targets.
>
>
>> Mechanisms must be put in place that allow to clean the dynamically 
>> create
>> artifacts between builds, thus allowing the process to be repeated 
>> without
>> having to perform "static" re-initialization.
>
> You mean, adding new projects between builds?

I meant rebuilding without the need to add all types, features, 
targets, etc.

Actually, the use-case I'm most interested in has three phases:

- Global initialisation: types, features, etc. Stuff that is rarely 
changed, namely in a "controlled" production environment.
- Project-specific initialisation: project and targets
- Build: virtual-targets, actual-targets

I want to be able to do the third-phase repeatedly as fast as possible.

(...)

>> ----------------
>> Porting strategy
>> ----------------
>> - Define a baseline for the initial effort. This can be done tagging 
>> the
>> BBv2 CVS at a certain point or doing checkouts by date (cvs co -D 
>> xxx).
>> Once the basic system is running, make regular merges of changes in 
>> BBv2.
>
> I think the next milestone can be such a basic.

When do you intend to release it? Do you plan many changes before that?
I'm planning on spending some time working on this now (e.g., this 
weekend), so I'll just use the version I have and hope that you don't 
change it too much :-)

> It might be better to stop any
> jam-V2 development until we figure out how Pythonisation goes on.

Excellent. I was considering merging diffs but that could turn out to 
be really unwieldy.

>> - Port the library bottom-up: select the basic functionality so it is 
>> ported
>> first; then, port the higher-level layers. See the file list at the 
>> bottom.
>
> This raises a fundamental question -- do we try to first rewrite 
> everything
> and then make all tests work, or do we rewrite *while* keeping all 
> tests
> working? The second approach seems more safe to me.
>
> To avoid breaking existing projects, we must make sure that most 
> existing
> modules are usable, say, user-config.jam can invoke 'feature.feature' 
> to
> declare a new feature. It means that Python modules must be something 
> exposed
> to bjam. Then, why don't we expose the Feature.py module to bjam as 
> soon as
> the module is written, so that we quickly detect any problems?

I meant rewrite everything and then make it work. Why?
The rational, test-oriented, left-hand part of my brain insists that we 
should keep everything working all the time.
However, I have a gut feeling that it will take too long to integrate 
bjam and Python in the way it is needed to do so.
I mean:
In order to use bjam to parse Jamfiles and directing the commands to 
the Python engine, a basic one way mechanism would-suffice (I may be 
missing something here, though).
OTOH, to perform a piecemeal transition to Python, one would need to be 
able to invoke functions from bjam to Python and vice-versa. Worse, we 
would need to be able to pass objects between them and manage their 
lifetimes.
I have no experience whatsoever with the internals of scripting 
languages, so I may be overwhelmed with just a minor issue!
For instance, I tried out the patch you sent (importing Python modules 
from within bjam) and, despite being so simple, it worked very well.
Proposal: would you be interested in trying out that integration while 
I keep on working on the Python port?

>> - How is this to be done? How/who will guarantee no overlappping 
>> occurs?
>
> You've started this project, so I'll be perfectly happy if you 
> coordinate the
> errort.

Ok. So, anyone willing to collaborate, please let me know so I make 
sure no one is duplicating the same work.

>
>> Then, in conditions convert:
>> if $(a)
>> {
>> ..
>> }
>>
>> to
>>
>> if len (a) > 0:
>
> Maybe,
>
> if not a
>
> ? Which wil catch both None and empty list.

Of course!

(...)

>> - Working repository: Boost CVS?
>
> Looks reasonable. On a branch?

Sure.

>
>> - Mailing list?
>
> The same? ;-)

I just asked because I didn't want to annoy people who are just using 
BB, let alone to scare off newcomers :-)

>> - Python version? SCons uses 1.5.2. I'd rather use at least 2.2 due 
>> to some
>> important features. Guidance please.
>
> 1.5.2 is way too old. I don't know exactly what changed between 2.2 
> and 2.3
> and 2.4, so have no option if 2.2 is better than 2.4 or not.

I don't either. Any Python expert around?

>> ------
>> Doubts
>> ------
>> - Scanners: not really sure how these work and their relation with 
>> bjam.
>> Guidance please.
>
> Basically, each jam target can be associated with a header scanning 
> rule. That
> rule is called when bjam decides to scan dependencies and adds 
> appropriate
> targets. Are you interested in some specific information?

I should have been clearer. My understanding is that scanners simply 
define the regex they are interested in and somehow pass it on to the 
build engine which does the actual scanning. Then, the scanner sets up 
the dependencies. Bottom line, the "hard work" is done by the build 
engine.

>> I tried to sort these in such a way that we can build a testable 
>> system
>> really soon.
>>
>> To port now:
>> ------------
>> build/feature.jam
>> build/property.jam
>> build/property-set.jam
>> build/type.jam
>> build/scanner.jam
>
> Maybe, move virtual-target.jam here? So that we can write simple 
> script to
> create a.o from a.cpp?

Ok.

>> build/toolset.jam
>
> This one definitely should be converted after virtual-target.jam -- 
> it's
> called from there.

AFAIK, it only calls actualize, so a mock-up could be use d for simple 
tests.

Best regards,

Pedro