Re: [jamboost] Wrapping up new design

From: Rene Rivera (grafik666_at_[hidden])
Date: 2002-01-01 17:43:15

On 2002-01-01 at 03:49 PM, david.abrahams_at_[hidden] (David Abrahams) wrote:

>Sorry if things have been a bit quiet lately. I'm just trying to keep
>all of these balls in the air at once...
>
>Anyway, I've been thinking about what's missing that would allow us to
>make rapid progress. It seems to me that we need a high-level view of
>how the build process will go. The rest is "just" detail.
>
>I've managed to write things down at the highest level here, but I am stuck
>on a few open questions which I'll bring up in a separate message to follow.

Hm, OK, I think I'll comment now and maybe raise some questions of my own :-)

And to setup my point of view here... I should say that I'm not using the
boost build for building boost itself. I'm using it to build my own somewhat
large project. So my take on things is based on how usefull, easy,
understandable things are from using this in the more general build system
case. Hopefully this doesn't intrude too much into boost itself, I hope :-]

>First, let's outline the constraints that come from the build
>tool. The main tasks of any build system using Jam are:
>
>1. Establish targets
>2. Generate build actions for the targets
> 2a. Attach variables to the targets which will be substituted into build
>actions
>3. Establish dependency relationships between the targets
>4. Establish target locations.
>
>I'll refer to steps 1, 2, and 3 as "building the dependency graph".
>There are also some details to handle header scanning, but the list
>above covers most of it.
>
>OK, now my vision for how the build system works:
>
>1. Jambase is read. BOOST_BUILD_PATH is set, so none of the default
> Jambase code takes effect. Instead, we search for the file called
> boost-build.jam on BOOST_BUILD_PATH and read that.
>
> ** the current Jambase gives an error about FTJam toolset
> definitions etc., if BOOST_BUILD_PATH is not set and the toolset
> definition is not set either. Probably that message should be
> extended to say something about setting BOOST_BUILD_PATH so that
> people are not confused. The FTJam toolset definitions aren't
> needed unless you're building Jam itself.

This raises one of my current displeasures with building with the current
system... having to set variables "outside" of the system. To me the ideal
would be for the system to figure out automatically where things are and how
to minimally operate without any upfront setup, other than
installing/extracting. And yes even if it's only one variable you have to set.
I would much rather see something like:

a. Change the name of the jam executable to something boost specific, maybe
"bjam" or "boostjam".
b. When invoked with the new name, have it search for some top level build
file from the current directory, and using that to find the "standard"
tools/build path.
c. Search through the BOOST_BUILD_PATH if set.
c. If that fails search for the build directory on some platform specific
location, i.e. where the build tool was installed.

This makes it so that it can work right out of the box, without any "user"
work.

>2. boost-build.jam first loads two modules, site-config.jam and
> user-config.jam. We can put empty versions in the Boost.Build
> directory so that there is no error if users don't put their own
> somewhere earlier in BOOST_BUILD_PATH (we'll re-use that trick a
> lot). The Actually, these should probably contain /LOTS/ of
> commented-out code which does basic configuration jobs. What do
> these things look like? For one thing, I'd like to avoid having the
> user set a bunch of obscure global variables. Let's do this through
> rule invocations (just meant to be an example, not neccessarily
> committed to this syntax):
>
> # ---- Sample site-config.jam file ----
> # loads the msvc module, which registers it as an available
> # toolset No special toolset location/configuration information
> # is given, so it is assumed that the toolset is already set up
> # (e.g. VCVARS32.BAT has been called), or that it's installed in
> # its standard location.
> using msvc ;
>
> # As above, but tells the system that we have two versions of the
> # gcc toolset installed, in the specified locations, with 2.95.3
> # being the default. The 'using' rule loads the given module and
> # calls its "configure" method with the rest of its
> # arguments. How a module treats configuration information, of
> # course, is up to the module.
> using gcc : 2.95.3 /usr/local/gcc-2.95.3
> 3.0.2 /usr/local/gcc-3.0.2
> ;
>
> # Same idea as above.
> using stlport : 4.5 ~/stlport-4.5
> 4.6b2 ~/stlport-4.6b2
> ;
>
> # does what ALL_LOCATE_TARGET currently does
> locate-built-targets bigdrive:/dave/builds ;

And again another saving outside setup work...

It would be nice to have things like site-config.jam and user-config.jam, or
more generally *config.jam work, mostly, without having per user intervention.
For me I like to setup all the build files for my developers ahead of time and
have them work for my project without them doing anything other than executing
a "build" command. This means that I do just about everything using relative
paths, or self discovery from the current directory. So more specifically it
would be nice to:

1. Not just have empty *config.jam files but usefull common defenitions of
them and have ways of overriding their settings/defenitions on a per
user/site/project basis.
2. Support relative paths for everything based on common locations. So the
ability to specify where stlport is located relative to the project root, and
having it correctly translated and used as a full path from that.

To do #1 I would suggest that it should be possible to do a "gcc.configure..."
after the "using gcc...". For example:

using gcc : 2.95.3 gcc-2.95.3
3.0.2 gcc-3.0.2
: <prefix>/usr/local
;
# Only resets the "standard" location of gcc tools.
gcc.configure : <prefix>/usr/local/gcc-tools ;

For stlport type tool additives there is one problem which I'm not sure how
boost developers tackle right now. For me I end up building stlport with each
different gcc version I'm using (2.96 & 3.0.3 right now, more later). I
created my own build script which deposits different "lib" directories in the
stlport directory, which happens to be someplace in my cvs tree. I end up with
"stlport/lib-gcc-2.96" and "stlport/lib-gcc-3.0.3" directories.

So supporting that in an orthagonal way to the stlport version itself would be
essential for me.

As for relative paths taking the configre after using type use it would be
nice to be able to do something like:

using stlport...
# Locate the stlport location relative to the project root. "@" is what
# you suggested for project naming stuff, so maybe that's not a good
# choice here.
stlport.configure 4.5 stlport : <prefix>@/../../sdk ;

>3. The system reads the Jamfile in the invocation directory. Jamfiles
> contain mostly /declarative/ rule invocations. Declarative rules
> build data structures describing targets (and other things,
> e.g. toolsets in toolset description files). Of course a Jamfile
> may also import modules and invoke rules that do more heavy-duty
> work. There are several important reason to use declarative rules
> in Jamfiles. First, the current system which constructs a
> dependency/action graph as each descriptive rule is read is quite
> inefficient for large projects. Secondly, we are unable to make any
> delayed decisions based on the entire contents of the jamfile. For
> example, we might want to do something completely different with
> the target descriptions, e.g. generate makefiles. Finally, it is
> too easy for the rules invoked and variables set by the user in a
> Jamfile to interfere with the build process.
>
>4. The system traverses the set of top-level targets and generates
> the dependency graph based on the expanded build description (the
> algorithm for expanding build descriptions is given at the bottom
> of
>
>http://www.crystalclearsoftware.com/cgi-bin/boost_wiki/wiki.pl?Boost.Build_D
>esign_Proposal).
> OK, so I've tossed off most of the work of the build system in one
> sentence. The rest of this document deals with that in more detail.
>
> The basic algorithm is as follows:
>
> Determine the subproject requirements
> For each main target in the Jamfile:
> Generate the targets which correspond to source files
> Generate the subvariant targets of the main target
> For each subvariant of the main target
> build the dependency graph between the source targets and
> the subvariant target
>
>Determining the subproject requirements
>
> This part is simple. The user tells us about the subproject
> using the following rules:
>
> project.project ( project-id : requirements * : default-build * )
>
> Declares a project or subproject. A subproject's id is a path,
> starting with the project id of which it is a subproject. The
> requirements and default build apply to any targets described in
> the Jamfile which do not explicitly declare others. A project
> rule invocation is mandatory in any Jamfile in a project which
> includes subprojects or uses other projects.
>
> project.jamfile-location( root-to-jamfile )
>
> Declares the location of this Jamfile with respect to the
> project root, in case the path given in the project rule does
> not describe the location of the jamfile.

Changing the name of the Jamfile would also be nice to have, especially for
platforms that need file extensions (win32). So maybe:

project.jamfile ( [ <name>jamfile-name ] [ <location>jamfile-location ] )

So one can then do:

project.jamfile <name>foobar.jam ;
...and...
project.jamfile <name>foobar.jam <location>build ;
...and...
project.jamfile <location>build ;

> project.source-location( root-to-source )
>
> Declares that relative paths in this Jamfile are all specified
> relative to the specified directory. Thus, a project with this
> structure:
>
> root
> +- build
> | `- Jamfile
> `- src
> +- foo.c
> `- bar.c
>
> might have the following Jamfile:
>
> project.project foobar ;
> project.jamfile-location build
> project.source-location src ;
>
> exe foobar : foo.c bar.c ;

Extreme suggestion here :-) The following would be cool to have:

project.source <location>src <pattern>*.c ;

# builds all the "c" files in the "src" directory.
exe foobar : * ;

...and/or...
exe foobar : foo bar ;

>Generating Targets
>
> Target names given by users in a Jamfile aren't, in general, the
> names of targets that will actually be used by the build system,
> since elements such as subvariant grist come into play. In order to
> keep the system usable, however, ungristed names are "claimed" by
> the system for NOTFILE targets which correspond to the user's
> notion of what should be built. When multiple subvariant builds
> have been requested these NOTFILE targets will depend on several
> built target files.
>
>....

Awaiting your next post with anticipation :-]

-- grafik - Don't Assume Anything
-- rrivera_at_[hidden] - grafik_at_[hidden]
-- 102708583_at_icq - Grafik666_at_AIM - Grafik_at_[hidden]