|
|
Boost-Build : |
From: David Abrahams (david.abrahams_at_[hidden])
Date: 2002-01-03 16:45:34
This is a followup message to the one below (on which I've also corrected the formatting of the 2nd diagram). I am going to try to treat some of the questions raised.
I like the idea raised at the end of using a toolset class with overridablerules to control what gets built. For the most part, that allows an arbitrarily fine level of control over intermediate targets and dependencies.
I also don't want to give the toolset too much power. There will be build jobs outside the domain of the toolset. For example, people will want rules to generate documentation, e.g. using TeX. I would like the tools for doingthese other jobs to be first-class citizens in the build system; they shouldn't be folded into the toolset description and they have a similar need for capability and configuration.
Lex/Flex/Yacc/Bison is an example of a sort of build tool which raises interesting problems, and thus may help us to discover the right approach. An abstract target description given by the user for an exectutable target might well include lex and yacc source, in addition to C, C++, Fortran, etc... These tools clearly need to be able to cooperate with a toolset to generate the dependency graph for some abstract targets. Literate programming tools which extract code from documentation might fall into a similar category.
I believe that the ultimate arbiter of control ought to be the type of target being generated. So, the toolset would get "first crack" at any abstracttarget that it declares it can generate (e.g. EXE, LIB, DLL, IMPLIB...). This allows us to deal simply with the simpler cases of, e.g., documentationgeneration which doesn't require the sort of "mixed intervention" described above for Lex/Yacc: The TeX module would declare. At that point, the question becomes a bit simpler: when it encounters a source file which it doesn't know how to handle,
--------------
This message is concerned with building the dependency graph. That job includes the generation of intermediate targets (e.g. when building an exe fromsources, the object files are intermediate) and the invoking of rules which establish dependency relationships and build actions.
I keep finding myself with the need to create new target types out of existing ones. For example, a python extension (PYD) is a kind of shared library. Conceptually, target types form a sort of inheritance hierarchy in my mind:
+--------+ +--------+ 'Link' +------------+
| source | 'Archive'..+ object +..............>| executable |
+----+---+ : +---+----+ : +------+-----+
| : | : |
+-------+----+ +----------+-+----------+ : +----+----+
| | | : | | V | |
+----------+ +--+--+Asm +--+--+ : +--+--+ +----+---+ +--+--+ +--+--+
| compiled | | ASM |...>| OBJ | :...>| LIB | | IMPLIB | | DLL | | EXE |
+----+-----+ +-----+ +-----+ +-----+ +--------+ +--+--+ +-----+
| ^ ^ ^ ^ |
+---+--+ : : : : +--+--+
| | : : : : | PYD |
+--+--+ +-+-+ : : : : +-----+
| CPP | | C +....:..........: :
+--+--+ +---+ 'C' : :
: : :
:................:..........:
'C++'
The ALL_CAPS names represent concrete target types. The dotted lines above represent rules that can be invoked to build one type of target from another.
Given a "main target" and a set of "source targets", a simple algorithm fordetermining the chain of targets and rules might be based on finding the shortest path in the above graph, traversing inheritance and build action links, from the source types to the target types. Some open questions:
1. Some rules combine multiple sources into a single target (e.g. Archive, Link). How does the algorithm decide when targets are combined?
2. It seems to me that in general, a toolset may change the dotted lines inthe above graph. The simplest example I can think of is that of Win32 compilers that use response files, resulting in the following:
+--------+ +--------+ Cmd +=====+ Link +------------+ | source | 'Archive'..+ object +....>[ RSP ].....>| executable | +----+---+ : +---+----++=====+ : +------+-----+ | : | : | +-------+----+ +----------+-+-------------+ : +----+----+ | | | : | | V | | +----------+ +--+--+Asm +--+--+ : +--+--+ +----+---+ +--+--+ +--+--+ | compiled | | ASM |...>| OBJ | :...>| LIB | | IMPLIB | | DLL | | EXE | +----+-----+ +-----+ +-----+ +-----+ +--------+ +--+--+ +-----+ | ^ ^ ^ ^ | +---+--+ : : : : +--+--+ | | : : : : | PYD | +--+--+ +-+-+ : : : :+-----+ | CPP | | C +....:..........: : +--+--+ +---+ 'C' : : : : : :................:..........: 'C++'
Other examples would be front-end C++ compilers that can't produce object files directly, but need to run some 'C' compiler as the back end. Should wetry to provide this flexibility in a general way, or does it make sense toask these toolsets to simply implement the rules in terms of sub-rules, aswe do now for response file handling? The latter approach keeps the basic graph structure fixed, but allows arbitrarily fine granularity to be introduced into each construction rule. I am not certain whether the limitation of a fixed graph structure is acceptable. Thoughts?
One thought might be to use the "class" facility in tools/build/new/class.jam to make each toolset a separate class with overridable methods which generate targets from sources. So, the "method" EXE<-source would be invoked to build an executable from sources. The default implementation would in turn invoke OBJ<-{type} where {type} is the concrete source type for each source file, and then Link to build the executable from the resulting objects. The Win32 response file toolset base class could implement the outer 'Link'rule by calling the 'Cmd' rule and the 'inner-link' rule (or something).
A scheme like this one ties the structure of the dependency graph to the toolset. I am not sure whether that's acceptable, since we're talking about allowing orthogonal features like STLPort and Python (with multiple version selection). Does it make sense for so much control to reside with the toolset, or do these other features need a way to intervene?
-Dave
===================================================
David Abrahams, C++ library designer for hire
resume: http://users.rcn.com/abrahams/resume.html
C++ Booster (http://www.boost.org)
email: david.abrahams_at_[hidden]
===================================================
------=_NextPart_000_0106_01C19476.10953430 Content-Type: text/html;
charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
Boost-Build list run by bdawes at acm.org, david.abrahams at rcn.com, gregod at cs.rpi.edu, cpdaniel at pacbell.net, john at johnmaddock.co.uk
<HTML><HEAD>
<META content="text/html; charset=iso-8859-1" http-equiv=Content-Type>
<META content="MSHTML 5.00.3103.1000" name=GENERATOR>
<STYLE></STYLE>
</HEAD>
<BODY>
<DIV><FONT size=2>This is a followup message to the one below (on which I've
also corrected the formatting of the 2nd diagram). I am going to try to treat
some of the questions raised.</FONT></DIV>
<DIV> </DIV>
<DIV><FONT size=2></FONT> </DIV>
<DIV><FONT size=2>I like the idea raised at the end of using a toolset class
with overridable rules to control what gets built. For the most part, that
allows an arbitrarily fine level of control over intermediate targets and
dependencies.</FONT></DIV>
<DIV> </DIV>
<DIV><FONT size=2>I also don't want to give the toolset too much power. There
will be build jobs outside the domain of the toolset. For example, people will
want rules to generate documentation, e.g. using TeX. I would like the tools for
doing these other jobs to be first-class citizens in the build system; they
shouldn't be folded into the toolset description and they have a similar need
for capability and configuration.</FONT></DIV>
<DIV> </DIV>
<DIV><FONT size=2>Lex/Flex/Yacc/Bison is an example of a sort of build tool
which raises interesting problems, and thus may help us to discover the right
approach. An abstract target description given by the user for an
exectutable target might well include lex and yacc source, in addition to C,
C++, Fortran, etc...</FONT><FONT size=2> These tools clearly need to
be able to cooperate with a toolset to generate the dependency graph for some
abstract targets. Literate programming tools which extract code from
documentation might fall into a similar category.</FONT></DIV>
<DIV> </DIV>
<DIV><FONT size=2>I believe that the ultimate arbiter of control ought tobe the
type of target being generated. So, the toolset would get "first crack" at any
abstract target that it declares it can generate (e.g. EXE, LIB, DLL,
IMPLIB...). This allows us to deal simply with the simpler cases of, e.g.,
documentation generation which doesn't require the sort of "mixed intervention"
described above for Lex/Yacc: The TeX module would declare. At that point, the
question becomes a bit simpler: when it encounters a source file which it
doesn't know how to handle, </FONT></DIV>
<DIV> </DIV>
<DIV><FONT size=2>--------------</FONT></DIV>
<DIV><FONT size=2>This message is concerned with building the dependency graph.
That job includes the generation of intermediate targets (e.g. when building an
exe from sources, the object files are intermediate) and the invoking of rules
which establish dependency relationships and build actions.</FONT></DIV>
<DIV> </DIV>
<DIV><FONT size=2>I keep finding myself with the need to create new target types
out of existing ones. For example, a python extension (PYD) is a kind of shared
library. Conceptually, target types form a sort of inheritance hierarchy inmy
mind:</FONT></DIV><PRE><FONT size=2> +--------+ +--------+ 'Link' +------------+
| source | 'Archive'..+ object +..............>| executable |
+----+---+ : +---+----+ : +------+-----+
| : | : |
+-------+----+ +----------+-+----------+ : +----+----+
| | | : | | V | |
+----------+ +--+--+Asm +--+--+ : +--+--+ +----+---+ +--+--+ +--+--+
| compiled | | ASM |...>| OBJ | :...>| LIB | | IMPLIB | | DLL | | EXE |
+----+-----+ +-----+ +-----+ +-----+ +--------+ +--+--+ +-----+
| ^ ^ ^ ^ |
+---+--+ : : : : +--+--+
| | : : : : | PYD |
+--+--+ +-+-+ : : : : +-----+
| CPP | | C +....:..........: :
+--+--+ +---+ 'C' : :
: : :
:................:..........:
'C++'
</FONT></PRE>
<DIV><FONT size=2>The ALL_CAPS names represent concrete target types. Thedotted
lines above represent rules that can be invoked to build one type of targetfrom
another. </FONT></DIV>
<DIV><FONT size=2> </FONT></DIV>
<DIV><FONT size=2>Given a "main target" and a set of "source targets", a
simple algorithm for determining the chain of targets and rules might be based
on finding the shortest path in the above graph, traversing inheritance and
build action links, from the source types to the target types. Some open
questions:</FONT></DIV>
<DIV> </DIV>
<DIV><FONT size=2>1. Some rules combine multiple sources into a single target
(e.g. Archive, Link). How does the algorithm decide when targets are
combined?</FONT></DIV>
<DIV> </DIV>
<DIV><FONT size=2>2. It seems to me that in general, a toolset may changethe
dotted lines in the above graph. The simplest example I can think of is that of
Win32 compilers that use response files, resulting in the
following:</FONT></DIV><PRE> </PRE><PRE> +--------+ +--------+ Cmd +=====+ Link +------------+ <BR> | source | 'Archive'..+ object +....>[ RSP ].....>| executable | <BR> +----+---+ : +---+----+ +=====+ : +------+-----+ <BR> | : | : | <BR> +-------+----+ +----------+-+-------------+ : +----+----+ <BR> | | | : | | V | | <BR> +----------+ +--+--+Asm +--+--+ : +--+--+ +----+---+ +--+--+ +--+--+<BR> | compiled | | ASM |...>| OBJ | :...>| LIB | | IMPLIB | | DLL | | EXE |<BR> +----+-----+ +-----+ +-----+ +-----+ +--------+ +--+--+ +-----+<BR> | ^ ^ ^ ^ | <BR> +---+--+ : : : : +--+--+ <BR> | | : : : : | PYD | <BR> +--+--+ +-+-+ : : : : +-----+ <BR> | CPP | | C +....:..........: : <BR> +--+--+ +---+ 'C' : : <BR> : : : <BR> :................:..........: <BR> 'C++' <BR> <BR></PRE>
<DIV><FONT size=2>Other examples would be front-end C++ compilers that can't
produce object files directly, but need to run some 'C' compiler as the back
end. Should we try to provide this flexibility in a general way, or does itmake
sense to ask these toolsets to simply implement the rules in terms of sub-rules,
as we do now for response file handling? The latter approach keeps the basic
graph structure fixed, but allows arbitrarily fine granularity to be introduced
into each construction rule. I am not certain whether the limitation of a fixed
graph structure is acceptable. Thoughts?</FONT></DIV>
<DIV> </DIV>
<DIV><FONT size=2>One thought might be to use the "class" facility in
tools/build/new/class.jam to make each toolset a separate class with
overridable methods which generate targets from sources. So, the "method"
EXE<-source would be invoked to build an executable from sources. The default
implementation would in turn invoke OBJ<-{type} where {type} is the concrete
source type for each source file, and then Link to build the executable from the
resulting objects. The Win32 response file toolset base class could implement
the outer 'Link' rule by calling the 'Cmd' rule and the 'inner-link' rule (or
something).</FONT></DIV>
<DIV> </DIV>
<DIV><FONT size=2>A scheme like this one ties the structure of the dependency
graph to the toolset. I am not sure whether that's acceptable, since we're
talking about allowing orthogonal features like STLPort and Python (with
multiple version selection). Does it make sense for so much control to reside
with the toolset, or do these other features need a way to
intervene?</FONT></DIV>
<DIV> </DIV>
<DIV><FONT size=2>-Dave</FONT></DIV>
<DIV> </DIV>
<DIV><FONT size=2>===================================================<BR>
David Abrahams, C++ library designer for hire<BR> resume: </FONT><A
href="http://users.rcn.com/abrahams/resume.html"><FONT
size=2>http://users.rcn.com/abrahams/resume.html></A></DIV>
<DIV> </DIV>
<DIV><FONT size=2> C++ Booster
(</FONT><A href="http://www.boost.org"><FONT
size=2>http://www.boost.org></A><FONT size=2>)
<BR> email: </FONT><A
href="mailto:david.abrahams_at_[hidden]"><FONT
size=2>david.abrahams_at_[hidden]</FONT></A><BR><FONT
size=2>===================================================<BR></FONT></DIV></BODY></HTML>
------=_NextPart_000_0106_01C19476.10953430--