Re: [Boost-build] prototyping alternative Boost.Build syntax

Subject: Re: [Boost-build] prototyping alternative Boost.Build syntax
From: Vladimir Prus (vladimir.prus_at_[hidden])
Date: 2016-10-28 11:10:29

On 28-Oct-16 5:40 PM, Stefan Seefeld wrote:

>>> OK, so b2 will - when compiled with --python - contain both a Jam
>>> language interpreter as well as a Python interpreter ?
>>
>> Yes.
>
> Thanks. (On a quick note on
> https://github.com/boostorg/build/wiki/Python: the instructions don't
> make it clear how this relates to Python at all. The "hello" directory
> contains a jamroot.jam file, and the instructions also mention the need
> to create a user-config.jam file. What does it take to make this example
> entirely replace Jam by Python, i.e. replace both jamroot.jam and
> user-config.jam by equivalent Python snippets / files ?)

It is presently not possible to replace jamroot.jam with a Python 
equivalent - precisely because it was not yet known what syntax
we can use.

>> is an object that can accept a set of properties and generate concrete
>> targets, which normally correspond to files. What you declare in
>> Jamfile is typically a metatarget.
>
> Ah, so a metatarget instantiates targets once features and other
> properties are fixed ?

Yes. The properties are not actually "fixed" though. Say, given

    exe a : a.cpp ;

and invocation of

    b2 toolset=gcc toolset=msvc

The metatarget corresponding to a will be asked to generate target 
twice, once with toolset=gcc and another as toolset=msvc as the set
of desired properties. The properties are not globally fixed.

>>> OK, so "exe" is a rule to build an executable target (file) from one or
>>> more sources ('a' from 'a.cpp' in the above case).
>>
>> We probably should should forget about word 'rule', since it has no
>> meaning for Python port really. exe is a function that creates a
>> metatarget that, when generated, tries to build executable file
>> from the sources provided.
>
> Hmm, I think I understand what you re saying. Still, I like the
> vocabulary of "rule", "target", "action" as the domain-specific
> vocabulary the entire industry has been using for decades. Of course,
> rules can be internally represented (implemented) by functions.

I agree that 'target' and 'action' are useful notions; there are mostly
preserved in Boost.Build, too.

I'm less sure about 'rule'. In make, 'rule' can be either ordinary rule 
or implicit rule. In b2, there are no implicit things - you pretty much
always specify the command-line to use to update a target, so I think
'rule' can be quite confusing.

>>>> In Python port with Python build description, it would be:
>>>>
>>>> - Boost.Build loads Jamfile.py (hopefully we'll find a better name),
>>>> arranging for 'exe' function to be in scope.
>>>
>>> I assume the "exe" rule is also defined in some Jam code snippet
>>> somewhere. How does b2 internally represent rules (and associated
>>> actions) ? I'd assume that internal representation doesn't depend on
>>> whether the definition comes from Jam or Python ?
>>
>> Again, let's only talk about Python port - and consider the different
>> between the current situation where 'exe' is present in Jamfile and
>> future possible situation when it will be used from Python build file.
>
> So who or what defines "exe" ?

Boost.Build has a global list of functions that are made available to 
any loaded Jamfile. 'exe' is in that list.

The way it ends up on this list is via type declarations. There's a 
mechanism, in type.py, so say, "declare a target type EXE". When this
happens, type.py invokes the following code in project.py (edited
for brevity)

     def add_rule_for_type(self, type):
         assert isinstance(type, basestring)
         rule_name = type.lower().replace("_", "-")

         def xpto (name, sources=[], requirements=[], default_build=[],
                   usage_requirements=[]):

             return self.manager_.targets().create_typed_target(
                 type, self.registry.current(), name, sources,
                 requirements, default_build, usage_requirements)

         self.add_rule(rule_name, xpto)

We convert 'EXE' to lowercase, and add an entry to that global registry,
with a name of 'exe' and the above function object as value.

>> 'exe' is just a name. It is bound to a Python function. Python functions
>> are represented in some way by Python interpreter. When the function is
>> called, it executes a single line of code:
>>
>>             return self.manager_.targets().create_typed_target(
>>                 type, self.registry.current(), name, sources,
>>                 requirements, default_build, usage_requirements)
>>
>> Here, 'type' comes from lexical closure, and will be 'EXE'. That line
>> of code creates a metarget, in this specific case instance of
>> TypedTarget class (targets.py). b2 knows nothing about it.
>
> I don't understand the above. I thought 'a' was the target, and the way
> to build it was by way of the 'exe' rule, which instructs b2 to generate
> 'a' by compiling 'a.cpp' ?? (As you can see, in my mind the distinction
> between 'rule' and 'target' is important.)

'exe' does not instruct b2 to create 'a' by compiling 'a.cpp'. It only
creates, in Python, a metatarget instance, which records the list of
sources, and the desired target type, here 'EXE'. When we generate
that metarget, it's only then we decide how transformation will work,
and it's only then we create Boost.Build VirtualTarget instances,
and only then we create b2 targets and actions.

>> Then, after Jamfile is read, TypedTarget.generate is called with some
>> properties. That produces a pile of instances of VirtualTarget class
>> (virtual_target.py). Then, VirtualTarget.actualize is called, and that
>> creates b2-level targets, which are then updated.
>
> This last paragraph makes it sound as if you use "virtual target" the
> way I use "rule".
> To me, the build system allows users to write a bunch of rules that
> express how targets are made. Then, when the build process is performed,
> some of the rules are picked based on feature and property values, and
> associated actions are scheduled. It sounds like you'd phrase that as
> "targets are created from metatargets". Yes ?

Let's look at make syntax:

     a: a.o
         gcc -o a a.o

Here, 'a' and 'a.o' are targets, 'gcc -o a a.o' is action, and the 
entire construct is 'rule'.

In Boost.Build,

    exe a : a.cpp ;

is metatarget. The 'a' and 'a.cpp' above are just names that metatarget
records, and they are not associated with filesystem, and there's no
command-line anywhere.

When metarget is generated, we generate 'ordinary targets' and 'ordinary 
actions', represented in Python code as VirtualTarget and Action 
classes. So we'd have

    - target bin/gcc/debug/a
    - target bin/gcc/debug/a.o
    - action "gcc -o a a.o"

There's no explicit concept of 'rule' anywhere - the first target refers 
to the action.

>> The 'exe' is a function that creates metatarget - all the objects that
>> it directly creates are normal Python objects, not something inside
>> b2.
>
> Ah, so b2 doesn't internally use a Python runtime ? If it contains a
> Python interpreter, but doesn't use a Python runtime, I assume it needs
> some translation phase that maps the Python runtime produced by the
> Python interpreter once it read in all the Python scripts into its own
> representation ?
>
> Where is that done in the b2 code ?

b2 exports a module called 'bjam' to Python code. So the final build
graph is created by Python code, in

   https://github.com/boostorg/build/blob/develop/src/build/engine.py

by the code like this:

   bjam_interface.call("set-update-action", self.action_name,
                                   targets, sources, [])

Therefore b2:

- Starts Python code
- Provides a very small interface to declare low-level targets and actions
- Does know know anything else about what Python code does.

-- 
Vladimir Prus
https://vladimirprus.com