Re: [jamboost] Eureka?

From: Vladimir Prus (ghost_at_[hidden])
Date: 2002-01-15 09:17:59

David Abrahams wrote:

> > > What is your concern about property relevance? The only thing it gets
> > > used for is determining target location.
> >
> > Because it also determinies the exact set of targets that are build and I
> > think it's important that boost system don't do any unnecessary actions.
>
> Yes, I realized that late last night.

Great!

> > Clearly, if you want to build PYD and no specific generator specific to
> > PYD is present, generator for DLL should be selected. How will it be given
> > precedence over <target-type-base>executable.
>
> Because the generator will be labelled with /all/ of the base properties to
> which it applies.

I understand it now.

> [snip] I'll insert my new stuff here:

> 1. FINDING THE GENERATOR SET
> ======= === ========= ===
>
> Generators will be chosen based on a set of build properties and the
> generator's matching criteria. A generator's matching criteria are composed
> of 3 lists:
>
> required-properties
> optional-properties
> rules
>
> and an optional "category". If no category is supplied, the generator has
> the empty category. For example, a generator for STLport might be in the
> <stdlib> category.

> The specificity of a match is given by the length of its match list.
> Basically, generators that match more properties will be more likely to be
> chosen. In each category with a matching generator is found, we select the
> generators with the longest match for the generator set.

I'm starting to feel like this might work indeed!

> 2. BUILD PROPERTY EXPANSION
> ===== ======== =========
>
> In this step, all generators get an opportunity to modify the build
> properties associated with the build request:
>
> For each generator in the generator set, call its "expand" rule to alter
> the properties in the build request. Most generators that can actually
> build targets will not want to implement expand; usually expand will only
> be used by generators that need to modify the build somehow, e.g. by adding
> #include paths. Note that the build property set is still one big wad
> available to all competing/interacting generators, so this would be an
> inappropriate place for a toolset generator to remove irrelevant
> properties.

Seems like you've omitted your proposal about selecting best match from each 
category. If this part is here, I think I agree. I think we should also state 
that expand rule may not modify non-free properties. OTOH, I don't think 
anybody will want to do it in any case. 

> 3. VIRTUAL TARGET GENERATION
> ======= ====== ==========
>
> In this step, each generator has an opportunity to build a representation
> of the virtual dependency graph for the requested target. By "virtual", I
> mean that the targets in the graph are objects with a record of their
> parents, children, build properties, etc., but that no DEPENDS calls or
> action rules have yet been invoked for them.
>
> For each generator in the generator set, call its "execute" rule. The
> "execute" rule should return a list of the virtual targets generated in its
> dependency graph. Generators that don't produce targets will return the
> empty list.

> Before returning from its execute rule, each generator may collect
> "synthesized" build properties from the sources and/or intermediate targets
> generated by the matching process it invoked.

Okay, no objections here.

> 4. VIRTUAL TARGET SELECTION
> ======= ====== =========
>
> In this step, we select one of the dependency graphs generated by a
> generator in the generator set. The criterion is simple: we choose the
> graph whose generator returned the shortest list of targets. 

Could you clarify: do you mean the minimum number of generated targets (seems 
like you've meant that before, or am I misreading you mail?), or minimum 
depth of the dependency graph? 

> In other
> words, we choose the shortest path from sources to targets. The root
> target(s) of the dependency graph are labelled with the generator, so that
> the generator for any virtual target can be retrieved.

> 5. ACTUAL TARGET GENERATION
> ====== ====== ==========
>
> In this step, we recursively descend the dependency graph, invoking the
> "finalize" rule of the generator associated with each target. In general,
> this will invoke action rules and DEPENDS to create the internal Jam
> dependency graph.

Okay, this two-phase process is probably close to what we're looking for.

> > Let me try some use cases.
> >
> > 1. STLPort
> > If there two generators (from STLport and STLnonport), which one will be
> > selected? I presume we can't run both?

Nevermind -- thinko. Probably, I had in my the question of in which order 
generators will be run.

> > 2. Consider my favourite transformation sequence:
> > asm.wd -> asm_parser.whl, asm_parser.dlp
> > asm_parser.whl -> asm_parser.cpp
> > asm_parser.dlp -> asm_parser.cpp
>
> I presume you mean asm_parser1.cpp/asm_parser2.cpp?

asm_lexer.dlp -> asm_lexer.cpp 
is the right transformation.

> > Both cpp a then compiled and linked.
>
> "are?"-----^

yes.

> foo
> / \
> a1.o a2.o
>
> a1.cpp a2.cpp
>
> a.whl a.dlp
> \ /
> asm.wd
>
> The biggest problem I see here is that the search won't proceed in the
> usual way. It will look like this just after having found .wd->.whl,.dlp :
>
> foo
>	|
> .o
>	|
> .cpp
>	|
> asm.whl asm.dlp
> \ /
> asm.wd

Why can't both whl -> cpp and dlp -> cpp tranformation be considered. Or 
either one can be prefered to another, randomly?

> Now what happens?
>
> In this very special case, it might be easier to handle this because .whl
> and .dlp both produce .cpp files, so the .cpp generator can deal with it.

Is this case that special. I guess antlr produces a couple of cpp files as 
well. How cpp generator can handle this situation? By hardcoding cases for 
*.wd, for *.g and, it effect, by looking the same way as 'Objects' rule?

> But suppose .dlp files generated .f77 files? I don't know how we'd deal
> with it.

Agh! Actaully, why can't we note that there are two pathes of equal length 
and.... wait! and what if transformation from .dlp to .obj requires 7 steps? 
Is it possible to handle it when search starts from source type? Hmm, I don't 
know how can be handle this case at all! Actually, this is the problem I was 
referring to in one of my previous emails:
"There is a question of what happens is a single action generates
several targets. I don't know the right answer yet."

> OTOH, I hope we can call this a degenerate case. 

Actually, it is the case of source which produces two files. I'm not sure if 
we can consider it rare.

> It is always possible to
> register highly-specialized generators which know about both their source
> and target types, and which can guide the process in between. So we might
> have a generator which can deal with .whl,.dlp->.o*, for example.

And it will be a generator with requirements (<target-type>OBJ 
<source-type>WD), which will just create a predefined graph. What I don't 
like here is that we abandom the general scheme -- we just specify what to do 
with particular source file type.

> > 3. What would you say about the idea of having special executable kind
>
> called
>
> > 'unit-test-EXE', which will do almost the same as 'EXE' but will run the
> > executable. Looks like all that is needed to the define transformation
>
> with
>
> > <target-type>unit-test-EXE?
>
> Well, that doesn't usually work well. You want a separate file on-disk
> which represents the test results. Otherwise, the executable is deleted by
> Jam whenever the test fails (and re-linked the next time you run the
> tests). See status/Jamfile for some pretty sophisticated testing rules...
>
> or maybe by "define the transformation with..." you mean that we register a
> generator with <target-type>unit-test-EXE which can generate from
> <target-type>EXE?

Yes, I mean a generator which will 
- call generator for <target-type>EXE
- so some magic which will allow executable to be run but not deleted in case 
of failure. On-disk file target can be added by this magic part as well.

> > Won't it be easy to create such a incorrect
> > transformation by mistake? I don't have any examples, but it scares me a
> > little.
>
> I hope not. Anyway, generator construction won't be for the uninitiated,
> and even then, we'll have a good set of default behaviors with hooks that
> help users to build well-behaved generators. At first, we'll want to write
> them "by hand", so we can figure out what the idioms are, but I'm sure
> we'll begin refactoring them pretty quickly.

Okay, after all this discussion my only remaining major concern is sources 
which generate several targets. I'm waiting for your comment on those.

- Volodya