Re: Why Jam?

From: kral123 (kral123_at_[hidden])
Date: 2002-02-21 05:14:04

--- In boost_at_y..., "bill_kempf" <williamkempf_at_h...> wrote:
> --- In boost_at_y..., "kral123" <kral123_at_y...> wrote:
> > You can provide them in source form and have a Makefile 
> > just like you do now for Jam.
> 
> I think you miss the point.  Jam is a very small executable, and I 
> don't even have to install it since I can set up a build process 
that 
> boost straps it with tools that already exist on my platform.  

You could use the programs I mentioned without installing, just as a 
bootstrap, too.  Most people could skip that step, as every 
development-capable system but Windows I can think of already has 
the required tools as part of the operating system installation.  

> There 
> is no native port of autoconf+automake that exists on my platform 
> (Windows), and so to use it I have to install much more then just 
the 
> autoconf/automake tools... I have to install the cygwin subsystem.
> The burden is too high.

I think you might be confused about what autoconf really is.  There 
are jam's configuration scripts and autotools' produced 
configuration scripts.  Both are scripts that need to be interpreted.

Jamfiles currently work on no existing systems, you need to install 
the interpreter first.  autotools' work on almost all 
development-capable systems except Windows out of the box.  For 
Windows, you need to install the interpreter first.

The interpreter for jamfiles is a custom program called jam, and 
further customized for boost usage.  The interpreter for autotools' 
is comprised of several programs (shell, textutils, fileutils) each 
of which does some part of the work.  The total size of the 
interpreter programs for autotools' configure scripts is probably 2 
megs whereas for Jam it is about 500k.

Autotools basically work in two layers, a developer-specific 
metadata layer that generates the configure file, and a 
user-specific layer (the configure file).  The user-specific layer 
does not require autoconf or automake.  Automake is a perl script so 
adds perl to the requirements for the developer-specific layer, 
which is also readily available in source and binary form as a 
native Windows app.

The set of jam script interpreting programs and the set of autotool 
script interpreting programs are all buildable as _native_ Windows 
applications.  Both interpreters can be provided in buildable source 
form along with boost and be used to bootstrap boost.

Both can build with native Windows compilers like VC++ 6.0 and 
Borland 5.

> Autotools isn't portable,

How is it not portable?  I'm personally using it on Windows 98, NT, 
2000, RedHat Linux, FreeBSD, and Solaris currently.  Some of the 
things I'm building are native Windows DLL files, .lib files, as 
well as applications for The WinCE PocketPC platform (These are 
built via cross-compilation from Windows and Linux which autotools 
makes supporting very easy).  If I was a geek, I could even run 
autotools' configure scripts _on_ my PocketPC PDA, as all the 
supporting tools can run on it.  It would be extremely pointless to 
try and develop physically on a PDA of course, but that it can be 
done right now today says something about portability, doesn't it? :)

If you mean autoconf itself doesn't support VC++ 6.0, Borland 5, 
etc. Windows compilers, you are also wrong.  It has in the main 
branch since 1999 for VC++, and I don't know when it first supported 
Borland but I've heard of people using it:

"Major changes in release 2.13:

* Support for building on Win32 systems where the only available C or
  C++ compiler is the Microsoft Visual C++ command line compiler
  (`cl').  Additional support for building on Win32 systems which are
  using the Cygwin or Mingw32 environments."

> and it's familiar only to Unix users/developers.

It's more familiar to non-Windows people as Windows people generally 
don't do any configuration management at all.  I'd say the concept 
of configuration management is more foreign in that space than 
autotools itself, as most Windows programmers have run into 
"./configure; make; make install" before.

There are a lot of projects (especially libraries similar in needs 
to boost) that are building their Windows programs via autotools, so 
it isn't as foreign as you might think.  Unfortunately, the majority 
of Windows developers still think providing a VC++ workspace file 
that you have to manually hack into a working condition for your 
host is acceptible.  This is an attitude that needs to be changed.  
The awareness of autotools for Windows needs to be raised.

> You're making this argument solely because it's 
> the tool you're familiar with.

No, I'm making it because I really want to use the boost thread 
library as it looks very well designed, but the current build system 
is so problematic that it doesn't work right on the platforms I use 
let alone the platforms I expect users to want to compile my code 
on, and it makes no attempt whatsoever to integrate itself properly 
with the host environment.  I have a hard enough time with boost's 
build system myself that I can't in good faith require users to put 
up with installing it and then manually passing all the information 
it requires (paths, libs, etc.) to any program that wants to link to 
boost stuff, so I stick to pthreads for now.

It just seems that the boost folk are reinventing the wheel with 
boost-Jam for no reason other than not fully understanding autotools 
and are coming up with a wheel that for many aspects, doesn't roll.  
I don't see why a new buildsystem that only works for a single 
project and only handles the build process and doesn't work with 
existing tools and requires a new interpreter everywhere is needed 
when autotools already does all you want, does more than boost-Jam 
is designed to do, is better understood, is more flexible, more 
portable, and will be updated to support the tools and architectures 
of tomorrow without requiring the involvement of the boost people.

> There's been a need for many years for a portable config/build 
> system, even for projects that use autoconf+automake that pretend 
to 
> be portable to non-Unix platforms,

Pretend?  That a silly thing to say when you see such projects 
running on everything from Desktop Windows systems to handheld WinCE 
devices to consoles like the Dreamcast and PlayStation 2, and 
cellphones without requiring any manual configuration whatsoever 
from the user.

Obviously it's C++ and there's no magic portability bullet possible, 
but by using autotools, you specify the requirements for your 
project rather than capabilities of the host, and this way new 
systems often simply 'just work' due to attention to portability put 
into the project in the past by developers.  For example, I use a 
publically available autoconf macro in my code that will 
automatically generate a standard project-local stdint.h file if one 
doesn't exist on the system 
(http://www.gnu.org/software/ac-archive/C_Support/
ac_need_stdint_h.html). 
It turned out lack of stdint.h is a pretty widespread thing, so that 
magic bullet one-liner in my configure.in 
(AC_NEED_STDINT_H(src/stdint.h)) saved me portability headaches on 
many systems.  The biggest bonus: I didn't have to write that macro, 
someone else did, and it benefitted thousands of developers rather 
than one project.

With autotools, massive numbers of people have already gone the same 
directions you are now, and have already dealt with all the 
problems, too.  It just seems nonsensical to throw out all that work 
and start over to produce something that you admit won't live up to 
autotools anyway, and won't fit your libraries into the proper 'way 
of thinking' on whatever host they are installed on (well, assuming 
some day boost does installation).  For instance, I want to be able 
to put "AM_PATH_BOOST" in my configure script and have all the 
include paths and libraries deduced and included by magic like with 
the rest of autotoolized libraries, and have the user be warned if 
they don't have boost and they should get it from 
http://www.boost.org if they don't have it.  See libxml2's 
AM_PATH_XML2 and the programs that use it for an example of how this 
mechanism works.

> and yet there's still no autoconf+automake for Windows that
> doesn't require cygwin or make assumptions about the system.

I just pointed out to you that you can use native Windows programs 
(mingw32 built shell, textutils, fileutils) to interpet configure 
scripts just like you use a native Windows program (jam.exe) to 
interpret jam scripts.  I don't see how Cygwin fits into this or 
what assumptions about the system are being made.  The code needed 
isn't Cygwin's.  Cygwin is a commercial product which consists of 
cygwin1.dll - a UNIX emulation layer, and existing GNU software 
compiled against that dll.

> I've searched high and low and have found not a single port of 
> autoconf+automake using mingw32.  Care to point one out?

autoconf+automake aren't things you port, they are portable scripts. 
 You port the interpreter, which in autoconf's case is the shell, 
textutils, and fileutils.  In automake's case it's perl.  If you are 
on the user side of the equation, you don't need autoconf, automake, 
or perl - only their output.

> > For an example, look at the toolchain created by the SDL people 
> that 
> > runs via mingw32 (http://www.libsdl.org/Xmingw32/).
> 
> I don't see a native port of autoconf+automake here, and 
> the "recommendation" is to use the cygwin environment.  In fact, 
the 
> page contains many shell scripts that can only run under cygwin.

I gave it as an example of a small Windows-native (mingw32) toolset 
that I know from experience can run configure scripts, as you seem 
to believe you need some goliath cygwin UNIX emulation installation 
just to run configure.  You don't, as shown.  The other info on that 
page is irrelevant, unless you plan on making realtime multimedia 
configuration management for boost with 3D sound and bumpmapping. ;)

> 10% of cygwin is still a LOT more then I can install.

Again, you're confused about needing Cygwin.  Cygwin is at the core 
cygwin1.dll, a UNIX portability layer that tries to make Windows 
feel like UNIX to applications built on it (for example, it 
translates paths used in open calls, treats things like "/dev/null" 
as special files, etc.).  Mingw32 is a layer that allows you to link 
POSIXish programs against the native Windows libraries and have them 
function like native Windows programs (for example, it uses native 
Windows paths in open calls).  Cygwin is an emulation layer and 
Mingw32 is a porting layer.

As for the size + installation, you won't need to install these 
programs if you do the bootstrapping method with them like you do 
now with jam.  The executable size isn't that different, 500k vs. 
around 2 megs, and unlike jam, most people won't need them anyway as 
they came with their operating system.

My reccomendations:

- Really consider using autotools.  I think a lot of the decision 
not to use autotools was based on a misunderstanding as to what they 
require and what they can do.

- Add the build support stuff for autotools outside of the main 
boost tree so those that don't need it, don't have to get it.  You'd 
have "boost" and "boost-support", two separate packages.  If you 
were on a Windows system and didn't already have the needed support, 
you'd first do something like "cd boost-support; ./configure 
--prefix=c:\tmp\boost-support; make; make install" set your path to 
include c:\tmp\boost-support, then build boost "cd ../boost; 
./configure; make; make install".  Very standard, very clean, and 
will support any future needs boost has.  You could wrap it into one 
command if you wanted.  Since boost reaches a lot of Windows users, 
maybe they'd pick up good habits after seeing such a system in use 
and start using configuration management in their own projects.

If you have any questions, feel free to email/respond.