From: Vladimir Prus (ghost_at_[hidden])
Date: 2002-01-15 09:17:59
David Abrahams wrote:
> > > What is your concern about property relevance? The only thing it gets
> > > used for is determining target location.
> > Because it also determinies the exact set of targets that are build and I
> > think it's important that boost system don't do any unnecessary actions.
> Yes, I realized that late last night.
> > Clearly, if you want to build PYD and no specific generator specific to
> > PYD is present, generator for DLL should be selected. How will it be given
> > precedence over <target-type-base>executable.
> Because the generator will be labelled with /all/ of the base properties to
> which it applies.
I understand it now.
> [snip] I'll insert my new stuff here:
> 1. FINDING THE GENERATOR SET
> ======= === ========= ===
> Generators will be chosen based on a set of build properties and the
> generator's matching criteria. A generator's matching criteria are composed
> of 3 lists:
> and an optional "category". If no category is supplied, the generator has
> the empty category. For example, a generator for STLport might be in the
> <stdlib> category.
> The specificity of a match is given by the length of its match list.
> Basically, generators that match more properties will be more likely to be
> chosen. In each category with a matching generator is found, we select the
> generators with the longest match for the generator set.
I'm starting to feel like this might work indeed!
> 2. BUILD PROPERTY EXPANSION
> ===== ======== =========
> In this step, all generators get an opportunity to modify the build
> properties associated with the build request:
> For each generator in the generator set, call its "expand" rule to alter
> the properties in the build request. Most generators that can actually
> build targets will not want to implement expand; usually expand will only
> be used by generators that need to modify the build somehow, e.g. by adding
> #include paths. Note that the build property set is still one big wad
> available to all competing/interacting generators, so this would be an
> inappropriate place for a toolset generator to remove irrelevant
Seems like you've omitted your proposal about selecting best match from each
category. If this part is here, I think I agree. I think we should also state
that expand rule may not modify non-free properties. OTOH, I don't think
anybody will want to do it in any case.
> 3. VIRTUAL TARGET GENERATION
> ======= ====== ==========
> In this step, each generator has an opportunity to build a representation
> of the virtual dependency graph for the requested target. By "virtual", I
> mean that the targets in the graph are objects with a record of their
> parents, children, build properties, etc., but that no DEPENDS calls or
> action rules have yet been invoked for them.
> For each generator in the generator set, call its "execute" rule. The
> "execute" rule should return a list of the virtual targets generated in its
> dependency graph. Generators that don't produce targets will return the
> empty list.
> Before returning from its execute rule, each generator may collect
> "synthesized" build properties from the sources and/or intermediate targets
> generated by the matching process it invoked.
Okay, no objections here.
> 4. VIRTUAL TARGET SELECTION
> ======= ====== =========
> In this step, we select one of the dependency graphs generated by a
> generator in the generator set. The criterion is simple: we choose the
> graph whose generator returned the shortest list of targets.
Could you clarify: do you mean the minimum number of generated targets (seems
like you've meant that before, or am I misreading you mail?), or minimum
depth of the dependency graph?
> In other
> words, we choose the shortest path from sources to targets. The root
> target(s) of the dependency graph are labelled with the generator, so that
> the generator for any virtual target can be retrieved.
> 5. ACTUAL TARGET GENERATION
> ====== ====== ==========
> In this step, we recursively descend the dependency graph, invoking the
> "finalize" rule of the generator associated with each target. In general,
> this will invoke action rules and DEPENDS to create the internal Jam
> dependency graph.
Okay, this two-phase process is probably close to what we're looking for.
> > Let me try some use cases.
> > 1. STLPort
> > If there two generators (from STLport and STLnonport), which one will be
> > selected? I presume we can't run both?
Nevermind -- thinko. Probably, I had in my the question of in which order
generators will be run.
> > 2. Consider my favourite transformation sequence:
> > asm.wd -> asm_parser.whl, asm_parser.dlp
> > asm_parser.whl -> asm_parser.cpp
> > asm_parser.dlp -> asm_parser.cpp
> I presume you mean asm_parser1.cpp/asm_parser2.cpp?
asm_lexer.dlp -> asm_lexer.cpp
is the right transformation.
> > Both cpp a then compiled and linked.
> / \
> a1.o a2.o
> a1.cpp a2.cpp
> a.whl a.dlp
> \ /
> The biggest problem I see here is that the search won't proceed in the
> usual way. It will look like this just after having found .wd->.whl,.dlp :
> asm.whl asm.dlp
> \ /
Why can't both whl -> cpp and dlp -> cpp tranformation be considered. Or
either one can be prefered to another, randomly?
> Now what happens?
> In this very special case, it might be easier to handle this because .whl
> and .dlp both produce .cpp files, so the .cpp generator can deal with it.
Is this case that special. I guess antlr produces a couple of cpp files as
well. How cpp generator can handle this situation? By hardcoding cases for
*.wd, for *.g and, it effect, by looking the same way as 'Objects' rule?
> But suppose .dlp files generated .f77 files? I don't know how we'd deal
> with it.
Agh! Actaully, why can't we note that there are two pathes of equal length
and.... wait! and what if transformation from .dlp to .obj requires 7 steps?
Is it possible to handle it when search starts from source type? Hmm, I don't
know how can be handle this case at all! Actually, this is the problem I was
referring to in one of my previous emails:
"There is a question of what happens is a single action generates
several targets. I don't know the right answer yet."
> OTOH, I hope we can call this a degenerate case.
Actually, it is the case of source which produces two files. I'm not sure if
we can consider it rare.
> It is always possible to
> register highly-specialized generators which know about both their source
> and target types, and which can guide the process in between. So we might
> have a generator which can deal with .whl,.dlp->.o*, for example.
And it will be a generator with requirements (<target-type>OBJ
<source-type>WD), which will just create a predefined graph. What I don't
like here is that we abandom the general scheme -- we just specify what to do
with particular source file type.
> > 3. What would you say about the idea of having special executable kind
> > 'unit-test-EXE', which will do almost the same as 'EXE' but will run the
> > executable. Looks like all that is needed to the define transformation
> > <target-type>unit-test-EXE?
> Well, that doesn't usually work well. You want a separate file on-disk
> which represents the test results. Otherwise, the executable is deleted by
> Jam whenever the test fails (and re-linked the next time you run the
> tests). See status/Jamfile for some pretty sophisticated testing rules...
> or maybe by "define the transformation with..." you mean that we register a
> generator with <target-type>unit-test-EXE which can generate from
Yes, I mean a generator which will
- call generator for <target-type>EXE
- so some magic which will allow executable to be run but not deleted in case
of failure. On-disk file target can be added by this magic part as well.
> > Won't it be easy to create such a incorrect
> > transformation by mistake? I don't have any examples, but it scares me a
> > little.
> I hope not. Anyway, generator construction won't be for the uninitiated,
> and even then, we'll have a good set of default behaviors with hooks that
> help users to build well-behaved generators. At first, we'll want to write
> them "by hand", so we can figure out what the idioms are, but I'm sure
> we'll begin refactoring them pretty quickly.
Okay, after all this discussion my only remaining major concern is sources
which generate several targets. I'm waiting for your comment on those.
Boost-Build list run by bdawes at acm.org, david.abrahams at rcn.com, gregod at cs.rpi.edu, cpdaniel at pacbell.net, john at johnmaddock.co.uk