Re: [Boost-build] feature, properties, variants, and all the rest

Subject: Re: [Boost-build] feature, properties, variants, and all the rest
From: Steven Watanabe (watanabesj_at_[hidden])
Date: 2017-08-07 17:25:14

AMDG

On 08/05/2017 08:43 PM, Stefan Seefeld via Boost-build wrote:
> On 08/05/2017 07:13 PM, Steven Watanabe via Boost-build wrote:
>> On 08/05/2017 04:27 PM, Stefan Seefeld via Boost-build wrote:
>>> On 08/05/2017 04:15 PM, Steven Watanabe via Boost-build wrote:
>>>> <snip>
>>>>   Why would you think that?  I just stated that
>>>> the engine only gets involved when the Jam targets
>>>> are passed to it.  (I'm considering the Jam interpreter
>>>> to be separate from the build engine here.)
>>> Ah, may be that's part of my misunderstanding. So let me outline my
>>> understanding of this. Please correct me as appropriate:
>>>
>>> The engine is a scheduler for actions needed to update targets. So it
>>> has a representation of targets to indicate whether they need updating,
>>> are up to date, or are somewhere in between. Actions are written in Jam,
>>> and are parsed into some IR by a Jam parser when the Jamfile is read in.
>>> At some point these actions are translated (by the Jam interpreter) into
>>> a sequence of commands that are run sequentially (by the scheduler).
>>>
>>   That's basically correct, except that you should
>> think of Jam as something like a general purpose
>> programming language.  The Jam parser, per se, doesn't
>> create the IR.  It just executes code.  The IR is
>> built by various rules defined by Boost.Build that
>> are called from the Jamfile.
> 
> I'm not sure I understand(, but I'm also not sure I have to). When the
> Jamfile is read in, the instructions it contains are parsed into some
> "IR", no ? And that IR is later interpreted / executed, no ?

  That's true, but this IR is more like a normal programming
language IR, like say LLVM.  It doesn't have any direct
correspondence with the dependency graph.  The main-target
level dependency graph and the virtual-target dependency graph
are both fully represented as Jam objects.  It's only at the
very end of the build process that the virtual targets
are actualized and the dependency graph is turned into
the form that's built-in to Jam.  Loading the Jamfile only
creates the main-target graph.

> Can you
> outline how that happens, and how the individual steps relate to rules
> and actions, and which of them are scheduled by the engine, and which
> are performed before engine is run (e.g., where in that process the jam
> targets are constructed) ?
> 

  Jam targets are created by virtual-target.actualize.  The
following code builds the Boost.System library at the
point where it is run:
# Some parameters omitted for the sake of clarity
local vtarget = [ targets.generate-from-reference /boost//system ] ;
local jam-targets = [ $(vtarget).actualize ] ;
UPDATE_NOW $(jam-targets) ;

This is what the top-level build process does, and
also what configure.check-target-builds does.

>>> (What I still don't quite understand is the relationship between rules
>>> and actions.)
>>>
>>   This is basically about low level Jam syntax.  Do
>> you really care about it or are you asking about something
>> different?
> 
> I do care to the degree that I want to understand b2's workflow, and how
> it relates to the one I'm in the process of designing, to see whether I
> am overlooking anything that ought to work but doesn't, in my own approach.
> 
>> <snip>
> 
> Who or what calls rules ?
> 

  Any jam code can call a rule.  Just like any
python code can call a python function.

>>> <snip>
>>> Ah, so it now looks like this happens after the Jamfile has been parsed,
>>> but before the engine is "started", and thus, before the config checks
>>> are run. (Thus the question: how can the config check results be used,
>>> if the lowering happens before they are known ?)
>>>
>>   When you run a config check, the subgraph corresponding
>> to the config check is immediately lowered and executed
>> (outside the normal order of execution).  This isn't really
>> the best abstraction, but it allowed us to patch support
>> for config checks into a system that originally had no
>> support for them at all.
> 
> I think you are a bit sloppy with terminology here: by "run a config
> check", are you referring to the corresponding line in the Jamfile ? or
> to some action performed by the scheduler, or something else ?

The target generation phase does it on demand.

> (This is
> really a critical point, as in my model I'm trying to have all "actions"
> (i.e., tasks that update artefacts) being orchestrated by the scheduler,
> while all the fabscripts do is populate the dependency graph with some
> initial artefacts and actions to update them.)
> 

  Every time Boost.Build needs to know the result
of a config check, it calls the scheduler.  The
scheduler gets called once for each config check,
and then a final time for the overall build.

>>   You can think of Boost.Build as using a pull model for
>> config checks, while you're using something more like a
>> push model.
> 
> OK. But when are these checks "pulled" ? They aren't performed while the
> Jamfile is parsed, but only a little later. What triggers this ?

  This happens when generating the dependency graph.  It happens
in each main target, after choosing the correct target alternative,
but before resolving dependencies.  config checks are just a special
case of a conditional property and they are resolved along with
all the other conditional properties, when computing the "common"
properties.

>>>> <snip>
>>>>   Because the dependencies are not explicitly modeled,
>>>> but are rather a result of the ordering imposed by
>>>> executing code it's impossible to schedule the
>>>> tasks in any way other than running them as they
>>>> are encountered.
>>> Ah. Yeah, that is a problem. But I suppose there would be ways to
>>> translate the "<use>A" expression into an additional dependency to A,
>>> thus forcing a particular order of execution ?
>>>
>>   That's too late.  By the time <use>A is applied, the configure
>> check has already been processed.
> 
> In the current b2 model, yes. But if config checks are handled like any
> other ordinary targets / artefacts, such "<use>A" statements could
> simply be translated into an additional dependency (in addition to
> connecting the corresponding features / properties), no ?
> 

  The definition of <use> in Boost.Build, is that
it takes the usage-requirements, but does not
create an actual dependency.  It should be possible
to make something similar work, however.

>>   All that the Boost.Build core
>> sees for a configure check is an opaque
>> <conditional>@some-internal-function-defined-by-check-target-builds
> 
> OK. Yes, that's how I do this right now, too: If "A" is a config check,
> "A.use" yields a "conditional" object that will expand into a feature
> set (either the "if_" argument or the "ifnot" argument used in the
> config check's construction, depending on whether it succeeded or not),
> and those will be merged into the feature set / requirements of the
> dependent artefact that contained that "use clause".
> 

  That's not the point I was trying to make.  The
conditional is completely opaque (maybe not in your
model, but in Boost.Build it is).  Boost.Build has
no idea, whatsoever, that it contains a config check.
It just runs the function that it's given.  If it
happens to run a config check that's well and good.
If it does something totally different, that's also fine.
It's impossible to create a dependency from this.

> So, again, I think it can work. But inconsistencies, cycles, and other
> error conditions may indeed be hard to detect and diagnose.
> 

Merging from the other sub-thread:
>>   Also, I don't know
>> how you handle #include scanning, but the way Jam
>> handles scanners causes cyclic #includes to create
>> a cyclic dependency in the internal nodes, which
>> is a total nightmare in the face of a dependency
>> graph that changes dynamically.
> 
> I haven't encountered any issues yet. The include-scanning-step is its
> own intermediate step that injects additional dependencies into the
> graph, and as long as they lead to existing files (which themselves
> don't need to be updated), I don't see issues. Where would that create
> problems ?

  Boost.Build can handle headers that are updated.
I glanced through your source, and I couldn't see
where you're handling the include-scanner.  Anyway,
Jam has a somewhat unusual #include scanner, in that
it scans each header separately, instead of scanning
per translation unit.

In Christ,
Steven Watanabe