RFC: On the Future of Jam, "FT Jam" and Boost

From: David Turner (david.turner_at_[hidden])
Date: 2001-06-29 07:01:22

Hello,

  I'd like to collect opinion from a large pool of Jam users
  regarding potential and upcoming enhancements to Jam. I'm
  sorry if the following is a bit long, but I've tried to
  make it as clear and accurate as possible.

  To summarize the following, I'd say that I propose the following:

     - to rename "FT Jam" to something a bit more pleasant

     - to create a SourceForge project for it, and use it for:

           * distribution os source and binary packages

           * providing a mailing list related to the
             developments / improvements made to the
             new "FT Jam" ( using the current list for
             normal Jam / Jamfile usage questions )

           * providing a CVS repository for the improved
             sources. This seems necessary for a lot of
             people who would like to contribute but do
             not master Perforce, nor want to take the
             time to install and learn it on their
             systems.

     - the project would contain two "modules":

           * the first one being the enhanced Jam sources
             themselves

           * the second one being the "boost" build
             system. it will depend on the first one.

  It's important to understand that all improvements integrated
  into "FT Jam" should be *completely* backwards compatible with
  the official Jam sources, in order to avoid breaking existing
  Jamfiles. As was said previously, some companies have made
  a tremendous investment in Jam.

  On the other hand, the boost build system will use some of
  the C sources provided by the enhanced Jam module, but
  also provide its own set of control files (i.e. the equivalent
  of "Jambase") as well as other C source files of its own.
  This should allow the creation of a single-executable
  "boost.exe".

  The goal of all of this is to be able to experiment nearly
  freely with new "features" in the "boost" module, while
  slowly moving the tested/validated ones into the "enhanced
  jam" one. The Boost control files will never migrate to
  this module though..

  I welcome any comments. More importantly, I welcome any
  suggestions for the new "enhanced jam" module. Please, please,
  do not suggest cryptic acronyms :-)

Best Regards,

- David Turner

====================================================================

  I. Jam and FT Jam:
  ------------------

  For those that do not know me, I have made some minor but
  critical improvements to the base Jam sources that are
  distributed under the name "FT Jam". You'll be able to
  find more information about it at the following address:

           http://www.freetype.org/jam/index.html

  I'd like to insist on the fact that the improvements
  present in this version of Jam are *completely* backwards
  compatible with the Jam 2.3.2 sources, and have been
  submitted to the Jam maintainer. They can be classified
  into two classes of improvements:

     - modifications to the C Jam sources, in order to
       run correctly on Win 9x systems, as well as implement
       new built-in rules (namely HDRMACRO and SUBST), etc..

     - modifications to the Jambase itself, in order to
       support more toolsets on Windows and OS/2 systems.

  In all cases, it's important to note that _existing_ Jamfiles
  should work without a single change with "FT Jam", and I'm
  commited to always meet this requirement in further improvements
  that could be made in the future. If you find something in
  FT Jam that seems to "break" one of your builds that otherwise
  work perfectly with the official release of Jam, please
  contact me to get this fixed !!

  By the way, I do not like the name "FT Jam", but couldn't
  find a better one for now. I welcome any suggestion to
  something more appealing (possibly avoiding strange
  acronyms, _please_, I like being able to spell my tools
  names in my basic english :-)

  II. Boost:
  ----------

  Recently, David Abrahams announced on this list the release
  of a new build system named 'Boost.Build', which I'll call
  'boost' in the rest of this document.

  Boost is a set of control files that over-ride the original
  Jambase and provide a different set of rules to developers 
  when they write "Boostfiles" (instead of Jamfiles).

  Boost manages advanced concepts that are completely alien
  to the original Jam/Jambase, like build variants,
  features/properties, requirements, etc.. and makes a
  professional developer's life a lot easier.

  The differences are so great that from a user point of
  view, Boost and Jam can even be thought as radically
  different designs.

  III. Jam limitations (wrt Boost):
  ---------------------------------

  Using Boost is currently a bit awkward for at least two
  reasons:

    - to use it, you need to copy several boost control
      files to your project's top directory. Since boost
      is still in rather heavy development, you need to
      update continously these files if you use them

    - you need to invoke Jam (or FT Jam) with the "-f"
      flag in order to not use the default Jambase

  Meanwhile, Boost is currently limited by some drawbacks
  of the original Jam design, and would benefit greatly from
  a few improvements made to the Jam C sources themselves.
  I have myself released a new version of FT-Jam recently
  that addressed one of these issues (while still maintaining
  compatibility, I insist !! :-)
  
  Because of these inconveniences, a recent proposal was
  made to fork the Jam sources in order to enhance Boost
  capabilities, while being able to build a single "boost"
  executable that would be, indeed, much easier to use than
  the current scheme.

  The problem with this approach is that improvements to
  one branch (e.g. Jam/FT Jam) would not benefit to the
  other one (respectively, the Boost version of the Jam
  sources).

  IV. Forking isn't necessary:
  ----------------------------

  After some thought, it seems however that we do not
  need to make a decision as drastic as forking the
  Jam source tree entirely.

  Since Boost is really a set of control "Jamfiles", the
  original Jam (or FT Jam) sources can be used _directly_,
  to build a single "boost.exe" that would incorporate
  all "Boostfiles". To explain this, I'll detail the way
  the Jam sources are currently organized:

     - a first set of C source files is used to create
       a library, called "libjam", that provides the
       base Jam functionality (i.e. control file parsing
       and execution).

     - a single control file, named "Jambase", contain default
       rule definitions for Jam, including "Cc", "Link",
       "Library", as well as various compiler-specific
       variable definitions and actions

     - the "mkjambase.c" program, used to convert a text
       file into a embeddable C source. It is currently
       used to convert "Jambase" into "jambase.c"

     - a front-end program named "jam.c" which is statically
       linked with "jambase.c" and "libjam", used to generate
       the single executable know as "jam.exe" or "jam"

  This scheme allows us to design Boost as the following:

     - a set of control files, like "all-your-base.jam",
       "features.jam", etc.. that can be processed through
       "mkjambase" in order to convert them to C source files

     - a front-end programmed, named "boost.c", which is
       statically linked with "all-your-base.c", etc.. and
       "libjam". It would be used to generate a single
       executable named "boost.exe" or "boost".

     - optionally, some other C source files used to augment
       "libjam" with new features (e.g. new rules).

  These two designs are not incompatible and allows boost to
  benefit from all improvements made to the Jam sources.

  V. Source Code Location :
  -------------------------

  The current Jam sources are currently available through the
  public Perforce Depot, (see //guests/david_turner/jam/src).

  Though I've submitted my changes more than one month ago,
  the Perfoce Jam team didn't find the time to review them
  to incorporate them into the official Jam sources (or simply
  reject them).

  I do not blame them for this, since they most probably have
  different priorities to deal with. However, as time passes,
  two things are happening:

    - I'm subject to add more and more enhancements
      to FT Jam, which only widens the gap between it and the
      official Jam sources (NB: while still preserving
      backwards compatibility). This will make the review
      and integration/rejection of FT Jam enhancements
      simply harder for the Jam team when they start
      doing it.

    - other people are starting to contribute changes, or
      willing to do so. Many of them are only familiar with
      CVS, and do not want to install or take the time to
      learn a new tool. I admit that I'm not really confident
      with Perforce myself, though I've tried rather hard :-(

  I thus propose to create a new CVS repository on a SourceForge
  project page to handle both the "FT Jam" and "Boost" sources.
  Using SourceForge has several benefits:

    - an easier management of access rights for different
      writers on the CVS repository than what can be done
      with the guests branch of the public Perforce depot
      (it seems).

    - the ability for _many_ developers to easily download
      the latest sources or stable releases through CVS,
      submit fixes, view revision history, etc..

    - the ability to parse the CVS sources from the web

    - a dedicated web page/address., plus download locations
      and information through HTTP/FTP.

  The current public depot sources will be kept as is, and
  updated periodically in order to submit only widely tested
  and stable enhancements, just in case the Jam team finds
  the time to review them..

  I know that Perforce is a lot better than CVS in a lot of
  points, especially for complex projects with lots of
  developers. However, I believe that using CVS for something
  as simple as the Jam+Boost sources should not hurt us.

Voilà !! Any comments on all of the above ??

- David Turner