Re: [jamboost] RFC: Proposal to port BBv2 to Python

From: Vladimir Prus (ghost_at_[hidden])
Date: 2005-01-20 08:16:19

Hi Pedro,

> my apologies for taking _way_ longer than I expected to post this: for
> a bunch of reasons I was unavailable for the last few weeks.
>
> I attach a draft of a porting strategy (Overview.txt) and the code I
> have already ported.
> Comments are very much appreciated.

Here they go:

> ----
> Goal
> ----
> Create a Python-based Boost.Build library.
> All the functionality available to BBv2 users shall be provided in the form
> of a library with a well-documented API. Parsers for higher-level languages,
> such as bjam, may be developed and use that API. 
> SCons shall be used as the build engine but its interface with the library
> shall be abstracted out, avoiding mixing library code with direct calls to a
> specific build engine, so that integration with other engines be possible
> and simple. 

I think it's better to split "Boost.Build in Python" and "Boost.Build in 
SCons" projects -- first convert to Python and only then to convert in SCons. 
We can't just take SCons and use it. First of all, we must decide how to best 
use it and if some of SCons design must be changed before we're able to use 
it. We must also understand all the features it provides as build engine to 
be sure we're using them all. And finally, I think that SCons needs some 
performance work, too.

So, switching to SCons is additional work. Maybe, we should first switch to 
Python?

> ---------
> Rationale
> ---------

[ snip quite reasonable explanations ]

> ---------------------
> Proposed Architecture
> ---------------------
> Unlike the current incarnation of Boost.Build, PyBB is supposed to deal with
> several consecutive builds on the same process. 
> For this reason, no global variables are allowed. The proposed approach to
> avoid this is to use a class - Boost.Build.Manager - that acts as a facade
> and is the root to access all data structures. 
> These include things such as a TypeRegistry, which contains all variables
> related to type registration that are global in BBv2 (See attached code, 
namely the Manager and TypeRegistry classes). 
> Similar classes should be developed for each main subsystem (e.g., features,
> targets, projects, etc.) 

I think that such design (no global data) will indeed be clearer and allow for 
better unit testing.

> PyBB needs not, and should not, have the concept of a Jamfile or similar.
> This would complicate its interface and add an additional burden for
> applications that do not use "project" files. 
> However, the grouping of targets in projects makes sense because it is used 
in most real world applications. In addition to this, associating names with 
projects make it easier to refer to other targets.
> 
> --------- ------------------- ---------
> | | 1 1 | | 1 * | |
> | |<----------->| ProjectRegistry |----------->| Project |
> | | | | | |
> | | ------------------- ---------
> | Manager | | 1
> | | |
> | | \|/ *
> | | ---------
> | | | |
> --------- | Target |
> | |
> ---------

One issue is that currently I plan to introduce per-project id-to-project 
mapping (so "/boost" can mean different things in different projects). Does 
this leave any functionality for the 'ProjectRegistry' class?

> All of the above denote what we could call "static" configuration, that is,
> it does not change in between several builds (provided that no more targets
> are added, of course). 
> Before invoking the build system, PyBB needs to transform targets (PyBB
> 'abstract' targets) in actual-targets (build system targets). In this 
process, it also creates virtual targets. 

> Mechanisms must be put in place that allow to clean the dynamically create
> artifacts between builds, thus allowing the process to be repeated without
> having to perform "static" re-initialization. 

You mean, adding new projects between builds?

> In order to abstract out the build system, an interface is defined that
> allows the engine to specify the actual targets, their dependencies, actions
> and substitution variables. 
> Actual implementations must be developed, e.g., for SCons.

Looks fine. 

> ----------------
> Porting strategy
> ----------------
> - Define a baseline for the initial effort. This can be done tagging the
> BBv2 CVS at a certain point or doing checkouts by date (cvs co -D xxx). 
> Once the basic system is running, make regular merges of changes in BBv2.

I think the next milestone can be such a basic. It might be better to stop any 
jam-V2 development until we figure out how Pythonisation goes on.

> - Port the library bottom-up: select the basic functionality so it is ported
> first; then, port the higher-level layers. See the file list at the bottom. 

This raises a fundamental question -- do we try to first rewrite everything 
and then make all tests work, or do we rewrite *while* keeping all tests 
working? The second approach seems more safe to me.

To avoid breaking existing projects, we must make sure that most existing 
modules are usable, say, user-config.jam can invoke 'feature.feature' to 
declare a new feature. It means that Python modules must be something exposed 
to bjam. Then, why don't we expose the Feature.py module to bjam as soon as 
the module is written, so that we quickly detect any problems?

> - How is this to be done? How/who will guarantee no overlappping occurs?

You've started this project, so I'll be perfectly happy if you coordinate the 
errort.

> Then, in conditions convert:
> if $(a)
> {
> ..
> }
> 
> to
> 
> if len (a) > 0:

Maybe, 

if not a

? Which wil catch both None and empty list.

> - Architectural docs: use the existing ones with the relevant updates?

Probably.

> ---------------
> Administrativia
> ---------------
> - Baseline for the intial port: CVS tag, checkout by date?

I think M11, when released, can be such baseline. All milestone releases are 
tagged.

> - Working repository: Boost CVS?

Looks reasonable. On a branch?

> - Mailing list?

The same? ;-)

> - Python version? SCons uses 1.5.2. I'd rather use at least 2.2 due to some
> important features. Guidance please. 

1.5.2 is way too old. I don't know exactly what changed between 2.2 and 2.3 
and 2.4, so have no option if 2.2 is better than 2.4 or not.

> ------
> Doubts
> ------
> - Scanners: not really sure how these work and their relation with bjam.
> Guidance please. 

Basically, each jam target can be associated with a header scanning rule. That 
rule is called when bjam decides to scan dependencies and adds appropriate 
targets. Are you interested in some specific information?

> I tried to sort these in such a way that we can build a testable system 
really soon.
> 
> To port now:
> ------------
> build/feature.jam
> build/property.jam
> build/property-set.jam
> build/type.jam
> build/scanner.jam

Maybe, move virtual-target.jam here? So that we can write simple script to 
create a.o from a.cpp?

> build/toolset.jam

This one definitely should be converted after virtual-target.jam -- it's 
called from there.

Thanks for your efforts,
Volodya