Re: [jamboost] Eureka?

From: David Abrahams (david.abrahams_at_[hidden])
Date: 2002-01-15 08:32:41

----- Original Message -----
From: "Vladimir Prus" <ghost_at_[hidden]>

> > What is your concern about property relevance? The only thing it gets
> > used for is determining target location.
>
> Because it also determinies the exact set of targets that are build and I
> think it's important that boost system don't do any unnecessary actions.

Yes, I realized that late last night.

> > > Are you sure this is good enough criteria? There's only one example
> > > of using optional properties in your email, and I'm not unsure about
> > > it.
> >
> > What do you mean by "I'm not unsure about it?"
>
> Either "not" or "un" should not have been left.

I'm not sure, either. In fact, I want to change it a bit. See later for
that, though.

> > These criteria handles things like target-type specificity. Properties
> > in an inheritance/refinement hierarchy can be composite properties
> > which expand to add properties for all of their bases. So for example,
> >
> > <target=type>PYD
> >
> > might expand to
> >
> > <target-type>PYD <target-type-base>PYD
> > <target-type-base>DLL <target-type-base>executable
> >
> > (see http://groups.yahoo.com/group/jamboost/message/88 if you need to
> > refer to the target-type refinement hierarchy). A generator which
> > wanted to match all executables might specify
> >
> > <target-type-base>executable
>
> Say there another generator, with
> <target-type-base>DLL
>
> Clearly, if you want to build PYD and no specific generator specific to
PYD
> is present, generator for DLL should be selected. How will it be given
> precedence over <target-type-base>executable.

Because the generator will be labelled with /all/ of the base properties to
which it applies.

pyd-generator requires:
<target-type-base>executable
<target-type-base>DLL
<target-type-base>PYD

dll-generator requires:
<target-type-base>executable
<target-type-base>DLL

> Or do you plan to always have
> default generator for each type, which generator will simply change target
> type to that of parent and invoke the matching process once again?

No, I planned to expand all of the bases so that a more-derived match would
always be a better one than a less-derived one.

> I see. And we can work without <source-type>, by means of selecting the
> generator with the less nonzero number of targets, right?

Yes, but then of course the generator has to succeed in generating a
complete graph for the target or it is ultimately discarded.

> > I wonder if this handles things like STLPort. Hmm, let's see: we want
> > the STLPort executable generator to be invoked preferentially to any
> > one toolset's generator. This generator will add <sysinclude> paths,
> > etc., to the build request, then re-invoke the process to generate the
> > executable.
> >
> > We can add
> >
> > <toolset> <target-type>EXE <target-type>DLL <target-type>IMPLIB
> > <target-type>LIB
> >
> > to its optional properties, and <target-type-base>executable to its
> > required properties. That should suffice.
>
> This is kind of magic to me :-) If you'll have another generator with
> required properties <toolset>gcc <target-type-base>executable, then will
> STLPort generator be selected becuse of extra <target-type> match?

I agree; that was bothering me, too. I'll insert my new stuff here:

I'm not too happy with the way this looks. It doesn't seem right to me that
something like STLPort support should have to do anything to get priority
over the toolset, in part because it doesn't "feel right", and in part
because it might break down. The toolset might well have registered a
generator which is specific enough to the build request that the STLPort
generator loses the competition. STLPort is really orthogonal to toolset
selection, so it should operate that way. I propose to modify the procedure
so that generators can declare a "category", and the best match from each
category gets a shot at the property set. Since I've now added several
wrinkles, I'll write out the process again:

Building A Target From Sources
======== = ====== ==== =======

Target declaration rules can of course be written to take the crude route of
directly building the dependency graph, but that's makes for a limited,
closed system. This describes how rules can be written which allow complex
interactions between orthogonal build properties (e.g. STLPort support,
Python support, toolset selection).

1. FINDING THE GENERATOR SET
======= === ========= ===

Generators will be chosen based on a set of build properties and the
generator's matching criteria. A generator's matching criteria are composed
of 3 lists:

required-properties
optional-properties
rules

and an optional "category". If no category is supplied, the generator has
the empty category. For example, a generator for STLport might be in the
<stdlib> category.

A generator doesn't match the build request unless all of its
required-properties are contained in the build request.

The matching process for a generator looks like this:

local match ;
if $(required-properties) in $(build-properties)
{
match = $(required-properties)
[ set.intersection optional-properties $(build-properties) ] ;
for local r in $(rules)
{
match = [ $(r) $(match) ] ; # maybe some other arguments, too
}
}
return match ;

The specificity of a match is given by the length of its match list.
Basically, generators that match more properties will be more likely to be
chosen. In each category with a matching generator is found, we select the
generators with the longest match for the generator set.

2. BUILD PROPERTY EXPANSION
===== ======== =========

In this step, all generators get an opportunity to modify the build
properties associated with the build request:

For each generator in the generator set, call its "expand" rule to alter the
properties in the build request. Most generators that can actually build
targets will not want to implement expand; usually expand will only be used
by generators that need to modify the build somehow, e.g. by adding #include
paths. Note that the build property set is still one big wad available to
all competing/interacting generators, so this would be an inappropriate
place for a toolset generator to remove irrelevant properties.

3. VIRTUAL TARGET GENERATION
======= ====== ==========

In this step, each generator has an opportunity to build a representation of
the virtual dependency graph for the requested target. By "virtual", I mean
that the targets in the graph are objects with a record of their parents,
children, build properties, etc., but that no DEPENDS calls or action rules
have yet been invoked for them.

For each generator in the generator set, call its "execute" rule. The
"execute" rule should return a list of the virtual targets generated in its
dependency graph. Generators that don't produce targets will return the
empty list.

At this point, the generator may collect a set of properties relevant to its
target construction method into a subvariant identifier. A database of
already-generated subvariant identifiers and their related targets can be
queried to see if the subvariant already exists. If it does, the generator
may use the cached data to return to its caller immediately.

To produce the dependency graph, a generator may well
invoke the matching and target calculation process again on some or all of
the source files, with the build property set changed to reflect the
generator's input target types. For example, a generator for executables
comes across a CPP file in a list of sources. It then replaces target types
in the build request with its list of input target types (OBJ LIB,...). The
matching process finds a generator which matches <target-type>OBJ with
optional <source-type>CPP. This generator, if eventually selected, is the
one that invokes the C++ compiler.

Why is CPP an <source-type>CPP an optional property for the C++ compiler?
Consider what happens when a a YPP source file appears in the list of
sources: the C++ generator should still be matched - it will want to invoke
the matching process itself once again, hopefully finding a generator which
matches <target-type>CPP/<source-type>YPP.

Why do we need <source-type>CPP at all? We don't: it's an optimization to
prevent less-specific generators from being invoked for build-property
expansion or virtual target generation.

Before returning from its execute rule, each generator may collect
"synthesized" build properties from the sources and/or intermediate targets
generated by the matching process it invoked.

4. VIRTUAL TARGET SELECTION
======= ====== =========

In this step, we select one of the dependency graphs generated by a
generator in the generator set. The criterion is simple: we choose the graph
whose generator returned the shortest list of targets. In other words, we
choose the shortest path from sources to targets. The root target(s) of the
dependency graph are labelled with the generator, so that the generator for
any virtual target can be retrieved.

5. ACTUAL TARGET GENERATION
====== ====== ==========

In this step, we recursively descend the dependency graph, invoking the
"finalize" rule of the generator associated with each target. In general,
this will invoke action rules and DEPENDS to create the internal Jam
dependency graph.

> Let me try some use cases.
>
> 1. STLPort
> Suppose we want <stdlib> property. Will STLport executable generator have
to
> consult the value of this property? If there's another standard library
> STLnonport, will generator for it have to consult the property as well. If
> there two generators (from STLport and STLnonport), which one will be
> selected? I presume we can't run both?

The <stdlib> property will have a value which indicates which generator to
choose:

<stdlib>STLPort or <stdlib>STLnonport or <stdlib>native

I'm pretty sure this was already clear, though. Am I misunderstanding your
question?

> 2. Consider my favourite transformation sequence:
> asm.wd -> asm_parser.whl, asm_parser.dlp
> asm_parser.whl -> asm_parser.cpp
> asm_parser.dlp -> asm_parser.cpp

I presume you mean asm_parser1.cpp/asm_parser2.cpp?
>
> Both cpp a then compiled and linked.
"are?"-----^
> exe foo : asm.wd ;
>
> When searching for transformations, the will be two pathes leading to
asm.wd.
> We'd need, during second pass (which computes targets &c), to make sure
> transformation from wd to whl and dlp won't be run two times.

foo
/ \
a1.o a2.o
| |
a1.cpp a2.cpp
| |
a.whl a.dlp
\ /
asm.wd

The biggest problem I see here is that the search won't proceed in the usual
way. It will look like this just after having found .wd->.whl,.dlp :

foo
|
.o
|
.cpp
|
asm.whl asm.dlp
\ /
asm.wd

Now what happens?

In this very special case, it might be easier to handle this because .whl
and .dlp both produce .cpp files, so the .cpp generator can deal with it.
But suppose .dlp files generated .f77 files? I don't know how we'd deal with
it.

OTOH, I hope we can call this a degenerate case. It is always possible to
register highly-specialized generators which know about both their source
and target types, and which can guide the process in between. So we might
have a generator which can deal with .whl,.dlp->.o*, for example.

> 3. What would you say about the idea of having special executable kind
called
> 'unit-test-EXE', which will do almost the same as 'EXE' but will run the
> executable. Looks like all that is needed to the define transformation
with
> <target-type>unit-test-EXE?

Well, that doesn't usually work well. You want a separate file on-disk which
represents the test results. Otherwise, the executable is deleted by Jam
whenever the test fails (and re-linked the next time you run the tests). See
status/Jamfile for some pretty sophisticated testing rules...

or maybe by "define the transformation with..." you mean that we register a
generator with <target-type>unit-test-EXE which can generate from
<target-type>EXE?

> So far I don't see any problem. I'm still unsure about the matching
> algorithm.

I hope you find today's installment improved.

> Can we insure that any particular transformation will be run
> first? Can poorly written transformation take precedence just because it
has
> a long set of optinal properties.

Of course, poorly-written code can be used to break any system.

> Won't it be easy to create such a incorrect
> transformation by mistake? I don't have any examples, but it scares me a
> little.

I hope not. Anyway, generator construction won't be for the uninitiated, and
even then, we'll have a good set of default behaviors with hooks that help
users to build well-behaved generators. At first, we'll want to write them
"by hand", so we can figure out what the idioms are, but I'm sure we'll
begin refactoring them pretty quickly.

-Dave