Re: [Boost-build] prototyping alternative Boost.Build syntax

Subject: Re: [Boost-build] prototyping alternative Boost.Build syntax
From: Stefan Seefeld (stefan_at_[hidden])
Date: 2016-10-28 11:31:06

On 28.10.2016 11:10, Vladimir Prus wrote:
>
>
> On 28-Oct-16 5:40 PM, Stefan Seefeld wrote:
>
>>>> OK, so b2 will - when compiled with --python - contain both a Jam
>>>> language interpreter as well as a Python interpreter ?
>>>
>>> Yes.
>>
>> Thanks. (On a quick note on
>> https://github.com/boostorg/build/wiki/Python: the instructions don't
>> make it clear how this relates to Python at all. The "hello" directory
>> contains a jamroot.jam file, and the instructions also mention the need
>> to create a user-config.jam file. What does it take to make this example
>> entirely replace Jam by Python, i.e. replace both jamroot.jam and
>> user-config.jam by equivalent Python snippets / files ?)
>
> It is presently not possible to replace jamroot.jam with a Python
> equivalent - precisely because it was not yet known what syntax
> we can use.

I understand. Perhaps that would be a nice goal for a Python prototype:
Rewrite that "hello" example to use Python for the build logic...

>
>>> is an object that can accept a set of properties and generate concrete
>>> targets, which normally correspond to files. What you declare in
>>> Jamfile is typically a metatarget.
>>
>> Ah, so a metatarget instantiates targets once features and other
>> properties are fixed ?
>
> Yes. The properties are not actually "fixed" though. Say, given
>
>    exe a : a.cpp ;
>
> and invocation of
>
>    b2 toolset=gcc toolset=msvc
>
> The metatarget corresponding to a will be asked to generate target
> twice, once with toolset=gcc and another as toolset=msvc as the set
> of desired properties. The properties are not globally fixed.

Ah, right.

>
>>>> OK, so "exe" is a rule to build an executable target (file) from
>>>> one or
>>>> more sources ('a' from 'a.cpp' in the above case).
>>>
>>> We probably should should forget about word 'rule', since it has no
>>> meaning for Python port really. exe is a function that creates a
>>> metatarget that, when generated, tries to build executable file
>>> from the sources provided.
>>
>> Hmm, I think I understand what you re saying. Still, I like the
>> vocabulary of "rule", "target", "action" as the domain-specific
>> vocabulary the entire industry has been using for decades. Of course,
>> rules can be internally represented (implemented) by functions.
>
> I agree that 'target' and 'action' are useful notions; there are mostly
> preserved in Boost.Build, too.
>
> I'm less sure about 'rule'. In make, 'rule' can be either ordinary
> rule or implicit rule. In b2, there are no implicit things - you
> pretty much
> always specify the command-line to use to update a target, so I think
> 'rule' can be quite confusing.

I'm not sure, but let's not get hung up on that. (Perhaps this is just a
naming thing, or perhaps it hints at something more important, I'll have
to think about it further...)

>
>>>>> In Python port with Python build description, it would be:
>>>>>
>>>>> - Boost.Build loads Jamfile.py (hopefully we'll find a better name),
>>>>> arranging for 'exe' function to be in scope.
>>>>
>>>> I assume the "exe" rule is also defined in some Jam code snippet
>>>> somewhere. How does b2 internally represent rules (and associated
>>>> actions) ? I'd assume that internal representation doesn't depend on
>>>> whether the definition comes from Jam or Python ?
>>>
>>> Again, let's only talk about Python port - and consider the different
>>> between the current situation where 'exe' is present in Jamfile and
>>> future possible situation when it will be used from Python build file.
>>
>> So who or what defines "exe" ?
>
> Boost.Build has a global list of functions that are made available to
> any loaded Jamfile. 'exe' is in that list.
>
> The way it ends up on this list is via type declarations. There's a
> mechanism, in type.py, so say, "declare a target type EXE". When this
> happens, type.py invokes the following code in project.py (edited
> for brevity)

Are 'type.py' and 'project.py' considered part of the 'engine' ?

>
>     def add_rule_for_type(self, type):
>         assert isinstance(type, basestring)
>         rule_name = type.lower().replace("_", "-")
>
>         def xpto (name, sources=[], requirements=[], default_build=[],
>                   usage_requirements=[]):
>
>             return self.manager_.targets().create_typed_target(
>                 type, self.registry.current(), name, sources,
>                 requirements, default_build, usage_requirements)
>
>         self.add_rule(rule_name, xpto)
>
> We convert 'EXE' to lowercase, and add an entry to that global registry,
> with a name of 'exe' and the above function object as value.
>
>>> 'exe' is just a name. It is bound to a Python function. Python
>>> functions
>>> are represented in some way by Python interpreter. When the function is
>>> called, it executes a single line of code:
>>>
>>>             return self.manager_.targets().create_typed_target(
>>>                 type, self.registry.current(), name, sources,
>>>                 requirements, default_build, usage_requirements)
>>>
>>> Here, 'type' comes from lexical closure, and will be 'EXE'. That line
>>> of code creates a metarget, in this specific case instance of
>>> TypedTarget class (targets.py). b2 knows nothing about it.
>>
>> I don't understand the above. I thought 'a' was the target, and the way
>> to build it was by way of the 'exe' rule, which instructs b2 to generate
>> 'a' by compiling 'a.cpp' ?? (As you can see, in my mind the distinction
>> between 'rule' and 'target' is important.)
>
> 'exe' does not instruct b2 to create 'a' by compiling 'a.cpp'. It only
> creates, in Python, a metatarget instance, which records the list of
> sources, and the desired target type, here 'EXE'. When we generate
> that metarget, it's only then we decide how transformation will work,
> and it's only then we create Boost.Build VirtualTarget instances,
> and only then we create b2 targets and actions.

ah, so b2 doesn't know about features and properties ? All that mapping
(and "metatarget instantiation") happens in the scripting layer ?

>
>>> Then, after Jamfile is read, TypedTarget.generate is called with some
>>> properties. That produces a pile of instances of VirtualTarget class
>>> (virtual_target.py). Then, VirtualTarget.actualize is called, and that
>>> creates b2-level targets, which are then updated.
>>
>> This last paragraph makes it sound as if you use "virtual target" the
>> way I use "rule".
>> To me, the build system allows users to write a bunch of rules that
>> express how targets are made. Then, when the build process is performed,
>> some of the rules are picked based on feature and property values, and
>> associated actions are scheduled. It sounds like you'd phrase that as
>> "targets are created from metatargets". Yes ?
>
> Let's look at make syntax:
>
>     a: a.o
>         gcc -o a a.o
>
> Here, 'a' and 'a.o' are targets, 'gcc -o a a.o' is action, and the
> entire construct is 'rule'.

right.
>
> In Boost.Build,
>
>    exe a : a.cpp ;
>
> is metatarget. The 'a' and 'a.cpp' above are just names that metatarget
> records, and they are not associated with filesystem, and there's no
> command-line anywhere.
>
> When metarget is generated, we generate 'ordinary targets' and
> 'ordinary actions', represented in Python code as VirtualTarget and
> Action classes. So we'd have
>
>    - target bin/gcc/debug/a
>    - target bin/gcc/debug/a.o
>    - action "gcc -o a a.o"
>
> There's no explicit concept of 'rule' anywhere - the first target
> refers to the action.

but there surely is the implied rule that connects the "first target" to
the other targets (the "sources") by virtue of the action that generates
the former from the latter ?!? Somewhere you have to hook up the "gcc
$(<) $(>)" action to the "target <- source" graph, no ?

>>> The 'exe' is a function that creates metatarget - all the objects that
>>> it directly creates are normal Python objects, not something inside
>>> b2.
>>
>> Ah, so b2 doesn't internally use a Python runtime ? If it contains a
>> Python interpreter, but doesn't use a Python runtime, I assume it needs
>> some translation phase that maps the Python runtime produced by the
>> Python interpreter once it read in all the Python scripts into its own
>> representation ?
>>
>> Where is that done in the b2 code ?
>
> b2 exports a module called 'bjam' to Python code. So the final build
> graph is created by Python code, in
>
>   https://github.com/boostorg/build/blob/develop/src/build/engine.py
>
> by the code like this:
>
>   bjam_interface.call("set-update-action", self.action_name,
>                                   targets, sources, [])
>
> Therefore b2:
>
> - Starts Python code
> - Provides a very small interface to declare low-level targets and
> actions
> - Does know know anything else about what Python code does.

Cool ! So what I'm after is that "small interface to declare low-level
targets and actions". Is that defined / documented somewhere ? What is
the code I need to read to understand that ?

Thanks,
        Stefan

-- 
      ...ich hab' noch einen Koffer in Berlin...