Re: [jamming] RFC: On the future of Jam, "FT Jam" and Boost

From: David Abrahams (david.abrahams_at_[hidden])
Date: 2001-06-29 08:49:49

----- Original Message -----
From: "David Turner" <david.turner_at_[hidden]>
> Hello,
>
>   I'd like to collect opinion from a large pool of Jam users
>   regarding potential and upcoming enhancements to Jam. I'm
>   sorry if the following is a bit long, but I've tried to
>   make it as clear and accurate as possible.
>
>   To summarize the following, I'd say that I propose the following:
>
>      - to rename "FT Jam" to something a bit more pleasant
>
>      - to create a SourceForge project for it, and use it for:
>
>            * distribution os source and binary packages
>
>            * providing a mailing list related to the
>              developments / improvements made to the
>              new "FT Jam" ( using the current list for
>              normal Jam / Jamfile usage questions )

If the boost build system finds wide use, it might make sense to use the
list you mention for that as well.

>            * providing a CVS repository for the improved
>              sources. This seems necessary for a lot of
>              people who would like to contribute but do
>              not master Perforce, nor want to take the
>              time to install and learn it on their
>              systems.

I think it might be unwise to start new projects at SourceForge, given the
following:
http://iwsun4.infoworld.com/articles/hn/xml/01/06/27/010627hnvalinux.xml?062

We at www.boost.org are currently investigating alternatives.

>   It's important to understand that all improvements integrated
>   into "FT Jam" should be *completely* backwards compatible with
>   the official Jam sources, in order to avoid breaking existing
>   Jamfiles. As was said previously, some companies have made
>   a tremendous investment in Jam.

Agreed.

>   By the way, I do not like the name "FT Jam", but couldn't
>   find a better one for now. I welcome any suggestion to
>   something more appealing (possibly avoiding strange
>   acronyms, _please_, I like being able to spell my tools
>   names in my basic english :-)

FWIW, I like "ftjam" and wouldn't waste time finding another name. I tried
to do the same with my python/c++ binding library for boost, but just ended
up with "the boost python library" (Boost.Python).

>   II. Boost:
>   ----------
>
>   Recently, David Abrahams announced on this list the release
>   of a new build system named 'Boost.Build', which I'll call
>   'boost' in the rest of this document.

This seems to happen every time something of interest outside the hardcore
C++ community appears at boost. People often refer to the boost python
library as simply "boost". I don't mind you calling Boost.Build just "boost"
as a kind of shorthand, as long as it's clear that boost has a very
different identity: www.boost.org is an open-source peer-reviewed C++
library development group.

>   Boost is a set of control files that over-ride the original
>   Jambase and provide a different set of rules to developers
>   when they write "Boostfiles" (instead of Jamfiles).

Except they're still called "Jamfile".

>   Boost manages advanced concepts that are completely alien
>   to the original Jam/Jambase, like build variants,
>   features/properties, requirements, etc.. and makes a
>   professional developer's life a lot easier.
>
>   The differences are so great that from a user point of
>   view, Boost and Jam can even be thought as radically
>   different designs.

Giving credit where it's due, the design of Boost.Build draws HEAVILY on the
design of Jam.

>   III. Jam limitations (wrt Boost):
>   ---------------------------------
>
>   Using Boost is currently a bit awkward for at least two
>   reasons:
>
>     - to use it, you need to copy several boost control
>       files to your project's top directory. Since boost
>       is still in rather heavy development, you need to
>       update continously these files if you use them

Just to clarify, these files don't have to live in your project's top
directory. For example, the boost Jamrules file currently contains:

BOOST_BUILD_INSTALLATION ?= $(TOP)$(SLASH)build ;
BOOST_BASE_DIR ?= $(BOOST_BUILD_INSTALLATION) ;

Which places these files in a subdirectory called "build". You could also
specify absolute paths which get them from some other location (e.g. a
server).

>     - you need to invoke Jam (or FT Jam) with the "-f"
>       flag in order to not use the default Jambase

That's quite inconvenient, and something I'd like to address.

>   Meanwhile, Boost is currently limited by some drawbacks
>   of the original Jam design, and would benefit greatly from
>   a few improvements made to the Jam C sources themselves.
>   I have myself released a new version of FT-Jam recently
>   that addressed one of these issues (while still maintaining
>   compatibility, I insist !! :-)

Thank you very much!

>   IV. Forking isn't necessary:
>   ----------------------------
>
>   After some thought, it seems however that we do not
>   need to make a decision as drastic as forking the
>   Jam source tree entirely.
>
>   Since Boost is really a set of control "Jamfiles", the
>   original Jam (or FT Jam) sources can be used _directly_,
>   to build a single "boost.exe" that would incorporate
>   all "Boostfiles". To explain this, I'll detail the way
>   the Jam sources are currently organized:
>
>      - a first set of C source files is used to create
>        a library, called "libjam", that provides the
>        base Jam functionality (i.e. control file parsing
>        and execution).
>
>      - a single control file, named "Jambase", contain default
>        rule definitions for Jam, including "Cc", "Link",
>        "Library", as well as various compiler-specific
>        variable definitions and actions
>
>      - the "mkjambase.c" program, used to convert a text
>        file into a embeddable C source. It is currently
>        used to convert "Jambase" into "jambase.c"
>
>      - a front-end program named "jam.c" which is statically
>        linked with "jambase.c" and "libjam", used to generate
>        the single executable know as "jam.exe" or "jam"
>
>   This scheme allows us to design Boost as the following:
>
>      - a set of control files, like "all-your-base.jam",
>        "features.jam", etc.. that can be processed through
>        "mkjambase" in order to convert them to C source files

Hmm; that's not what I envisioned.

1. allyourbase.jam is really a modified Jambase. For a while I tried to
ensure that it would be strictly compatible with the original Jambase, but
eventually gave up. Still, as long as users' Jamfiles stay away from using
variables with certain naming conventions (I'm thinking of names like
"gALL_UPPER_CASE") I think it should be possible to roll the changes back
into the Jambase from FTJam without breaking any code. There are two issues:

  a. The original Jambase would cause an error if you didn't set variables
describing your single toolset. That behavior is inappropriate for
potentially multi-compiler builds.

  b. The original Jambase rules are underspecified and there are no unit
tests for them, so it's hard to ensure that you've preserved the intended
behavior. We could deal with this by writing improved specification and unit
tests for the Jambase rules we modify.

2. I /like/ the fact that features.jam and the toolset definitions are not
compiled into the Jam executable. It keeps the build system configurable and
customizable. It should be possible to add features, toolsets, and variants
without recompiling the executable.

>      - a front-end programmed, named "boost.c", which is
>        statically linked with "all-your-base.c", etc.. and
>        "libjam". It would be used to generate a single
>        executable named "boost.exe" or "boost".

Back to the naming issue: it's a small thing, but I wouldn't like to
distribute an executable called "boost" unless the www.boost.org
participants agreed that it was appropriate.

>      - optionally, some other C source files used to augment
>        "libjam" with new features (e.g. new rules).
>
>   These two designs are not incompatible and allows boost to
>   benefit from all improvements made to the Jam sources.

All that said, I like the general direction you're going in.

>   V. Source Code Location :
>   -------------------------

<snip>

>   I thus propose to create a new CVS repository on a SourceForge
>   project page to handle both the "FT Jam" and "Boost" sources.
>   Using SourceForge has several benefits:
>
>     - an easier management of access rights for different
>       writers on the CVS repository than what can be done
>       with the guests branch of the public Perforce depot
>       (it seems).
>
>     - the ability for _many_ developers to easily download
>       the latest sources or stable releases through CVS,
>       submit fixes, view revision history, etc..
>
>     - the ability to parse the CVS sources from the web
>
>     - a dedicated web page/address., plus download locations
>       and information through HTTP/FTP.

I'd like to suggest that you consider hosting FT Jam where boost ends up
being hosted, since we are currently exploring another CVS host with better
long-term prospects. The other advantage to this is that we anticipate
having the ability to perform server-side maintenance jobs, such as moving
files in the repository, etc., for which you currently have to petition (the
almost certainly understaffed) SourceForge.

-Dave