Re: Why Jam?

From: bill_kempf (williamkempf_at_[hidden])
Date: 2002-02-21 10:33:53

--- In boost_at_y..., "kral123" <kral123_at_y...> wrote:
> --- In boost_at_y..., "bill_kempf" <williamkempf_at_h...> wrote:
> > --- In boost_at_y..., "kral123" <kral123_at_y...> wrote:
> > > You can provide them in source form and have a Makefile 
> > > just like you do now for Jam.
> > 
> > I think you miss the point.  Jam is a very small executable, and 
I 
> > don't even have to install it since I can set up a build process 
> that 
> > boost straps it with tools that already exist on my platform.  
> 
> You could use the programs I mentioned without installing, just as 
a 
> bootstrap, too.  Most people could skip that step, as every 
> development-capable system but Windows I can think of already has 
> the required tools as part of the operating system installation.  
> 
> > There 
> > is no native port of autoconf+automake that exists on my platform 
> > (Windows), and so to use it I have to install much more then just 
> the 
> > autoconf/automake tools... I have to install the cygwin subsystem.
> > The burden is too high.
> 
> I think you might be confused about what autoconf really is.  There 
> are jam's configuration scripts and autotools' produced 
> configuration scripts.  Both are scripts that need to be 
interpreted.

Actually, I'm not.

> Jamfiles currently work on no existing systems, you need to install 
> the interpreter first.  autotools' work on almost all 
> development-capable systems except Windows out of the box.  For 
> Windows, you need to install the interpreter first.
> 
> The interpreter for jamfiles is a custom program called jam, and 
> further customized for boost usage.  The interpreter for autotools' 
> is comprised of several programs (shell, textutils, fileutils) each 
> of which does some part of the work.  The total size of the 
> interpreter programs for autotools' configure scripts is probably 2 
> megs whereas for Jam it is about 500k.

Requiring a shell is, by far, the worst culprit here, but the fact of 
the matter is, there's no native port of most of the parts required 
for autoconf.  You have to install cygwin to have the needed parts.

Frankly, though, even if there were a native port of all the stuff 
required for autoconf you'd still be talking about at least 2 megs of 
stuff that has to be installed.  Boost strapping something that large 
isn't really an option as far as I'm concerned, and again the powers 
that be will have problems with an installation that large.
 
> Autotools basically work in two layers, a developer-specific 
> metadata layer that generates the configure file, and a 
> user-specific layer (the configure file).  The user-specific layer 
> does not require autoconf or automake.  Automake is a perl script 
so 
> adds perl to the requirements for the developer-specific layer, 
> which is also readily available in source and binary form as a 
> native Windows app.

The perl ports I'm aware of are intrusive installations (as they 
probably should be), which the powers that be won't allow.  I know, 
I've looked into using python.  I'm lucky enough (for now) to be able 
to install it on my personal box simply because (temporarily at 
least) I'm exempt from the monitoring that most in this company are 
subject to.  But I couldn't extend this to include the build server.  
Based on feedback on here, I'm not the only one in this situation.

You can argue as much as you want to, but the _reality_ is that I 
can't use autoconf+automake, but I can get away with Jam.
 
> The set of jam script interpreting programs and the set of autotool 
> script interpreting programs are all buildable as _native_ Windows 
> applications.  Both interpreters can be provided in buildable 
source 
> form along with boost and be used to bootstrap boost.

I'm a Boost developer, so I'd have to have access to ALL of 
autoconf+automake, and no, not all of the tools required for that 
exist as native Windows applications.  Even if they did, again, I 
can't install them.
 
> Both can build with native Windows compilers like VC++ 6.0 and 
> Borland 5.
> 
> > Autotools isn't portable,
> 
> How is it not portable?

It requires a cygwin to run on Windows... that classifies it as non-
portable IMHO.

> > and it's familiar only to Unix users/developers.
> 
> It's more familiar to non-Windows people as Windows people 
generally 
> don't do any configuration management at all.  I'd say the concept 
> of configuration management is more foreign in that space than 
> autotools itself, as most Windows programmers have run into 
> "./configure; make; make install" before.

There is less of a need for it on Windows, true. But this doesn't 
change anything about what I said.
 
> There are a lot of projects (especially libraries similar in needs 
> to boost) that are building their Windows programs via autotools, 
so 
> it isn't as foreign as you might think.  Unfortunately, the 
majority 
> of Windows developers still think providing a VC++ workspace file 
> that you have to manually hack into a working condition for your 
> host is acceptible.  This is an attitude that needs to be changed.  
> The awareness of autotools for Windows needs to be raised.

*sighs*  I'm a professional Windows programmer.  I've used 
autotools.  I've even used it on Windows platforms.  My level of 
awareness is very high.  I don't think I'm unique, or even in a small 
elite class of developers, in this regard.  Yet I'm also in a very 
large group of developers who *can't* use autotools every where.  
Autotools was a good idea, but there are too many choices that were 
made that make it a poor choice for many of us.  At this point I 
don't think I can convince you of this, but whether or not I can 
doesn't change the reality I must live with.

> > You're making this argument solely because it's 
> > the tool you're familiar with.
> 
> No, I'm making it because I really want to use the boost thread 
> library as it looks very well designed, but the current build 
system 
> is so problematic that it doesn't work right on the platforms I use 
> let alone the platforms I expect users to want to compile my code 
> on, and it makes no attempt whatsoever to integrate itself properly 
> with the host environment.  I have a hard enough time with boost's 
> build system myself that I can't in good faith require users to put 
> up with installing it and then manually passing all the information 
> it requires (paths, libs, etc.) to any program that wants to link 
to 
> boost stuff, so I stick to pthreads for now.

The only missing piece to the build process right now is 
installation.  That's being worked on.  And if that's the only thing 
preventing you from using it, shouldn't it be beyond trivial for you 
to write your own script to do the installation?

As for the problems with building... that's something that should be 
brought to the attention of the developers so that the problems can 
be fixed.  I use Jam daily, and it's probably the easiest build 
system I've ever used (well, besides hitting the compile button in my 
IDE), so I'm more then a little curious about what problems you've 
encountered.

> It just seems that the boost folk are reinventing the wheel with 
> boost-Jam for no reason other than not fully understanding 
autotools 
> and are coming up with a wheel that for many aspects, doesn't 
roll.  

You're reaching the wrong conclusion.  A very big portion of the 
Boost community are developers working on *nix boxes who are very 
familiar with autotools.  Others, such as myself, may have some 
Windows biases, but we are familiar with autotools none-the-less, and 
might even be willing to use them (I say might, because as a 
developer I personally find autotools to be a pain in the neck), but 
for reasons out of our control, we can't.  Don't assume the Boost 
community is ignorant or stupid.

> I don't see why a new buildsystem that only works for a single 
> project and only handles the build process and doesn't work with 
> existing tools and requires a new interpreter everywhere is needed 
> when autotools already does all you want, does more than boost-Jam 
> is designed to do, is better understood, is more flexible, more 
> portable, and will be updated to support the tools and 
architectures 
> of tomorrow without requiring the involvement of the boost people.

Again, most of your claims are true only for *nix boxes and users.  
And your claims about Jam aren't really accurate either... there's a 
huge body of developers familiar with it, they use it on very large 
projects that must be built on many platforms.  It's extremely 
flexible, and is much easier to use then autotools, at least IMHO.  
You're right that autotools does more, because Jam is solely a make 
replacement and doesn't address configuration issues, but that's not 
enough of an argument here.

> > There's been a need for many years for a portable config/build 
> > system, even for projects that use autoconf+automake that pretend 
> to 
> > be portable to non-Unix platforms,
> 
> Pretend?  That a silly thing to say when you see such projects 
> running on everything from Desktop Windows systems to handheld 
WinCE 
> devices to consoles like the Dreamcast and PlayStation 2, and 
> cellphones without requiring any manual configuration whatsoever 
> from the user.

Again, give me a full autotools installation on Windows that doesn't 
rely on cygwin.  Even better yet, one that doesn't require another 
shell.
 
> Obviously it's C++ and there's no magic portability bullet 
possible, 
> but by using autotools, you specify the requirements for your 
> project rather than capabilities of the host, and this way new 
> systems often simply 'just work' due to attention to portability 
put 
> into the project in the past by developers.  For example, I use a 
> publically available autoconf macro in my code that will 
> automatically generate a standard project-local stdint.h file if 
one 
> doesn't exist on the system 
> (http://www.gnu.org/software/ac-archive/C_Support/
> ac_need_stdint_h.html). 
> It turned out lack of stdint.h is a pretty widespread thing, so 
that 
> magic bullet one-liner in my configure.in 
> (AC_NEED_STDINT_H(src/stdint.h)) saved me portability headaches on 
> many systems.  The biggest bonus: I didn't have to write that 
macro, 
> someone else did, and it benefitted thousands of developers rather 
> than one project.

I understand this, but it doesn't (can't) change my position.
 
> With autotools, massive numbers of people have already gone the 
same 
> directions you are now, and have already dealt with all the 
> problems, too.  It just seems nonsensical to throw out all that 
work 
> and start over to produce something that you admit won't live up to 
> autotools anyway, and won't fit your libraries into the proper 'way 
> of thinking' on whatever host they are installed on (well, assuming 
> some day boost does installation).  For instance, I want to be able 
> to put "AM_PATH_BOOST" in my configure script and have all the 
> include paths and libraries deduced and included by magic like with 
> the rest of autotoolized libraries, and have the user be warned if 
> they don't have boost and they should get it from 
> http://www.boost.org if they don't have it.  See libxml2's 
> AM_PATH_XML2 and the programs that use it for an example of how 
this 
> mechanism works.

I never claimed that the Boost.Build system wouldn't every live up to 
autotools.  That's actually a silly thing for you to try and claim 
that I said.  I also understand the feeling that "reinventing the 
wheel" is wasted effort, but there are times when you have to, and 
other times when you're better off doing so.  Autotools is showing 
enough problems that even amongst it's ardent users there's a desire 
to replace it with the next generation solution.  It's called 
progress.  I'm not going to claim that Boost.Build will do this, but 
I'm also not going to claim that it won't.  In the mean time, 
however, it's making my life easier and solving a problem for which 
autotools can't.

> > and yet there's still no autoconf+automake for Windows that
> > doesn't require cygwin or make assumptions about the system.
> 
> I just pointed out to you that you can use native Windows programs 
> (mingw32 built shell, textutils, fileutils) to interpet configure 
> scripts just like you use a native Windows program (jam.exe) to 
> interpret jam scripts.  I don't see how Cygwin fits into this or 
> what assumptions about the system are being made.  The code needed 
> isn't Cygwin's.  Cygwin is a commercial product which consists of 
> cygwin1.dll - a UNIX emulation layer, and existing GNU software 
> compiled against that dll.

You haven't pointed me to a native install of autotools yet.  
Further, the inclusion of a shell means it's not native, now doesn't 
it?
 
> > I've searched high and low and have found not a single port of 
> > autoconf+automake using mingw32.  Care to point one out?
> 
> autoconf+automake aren't things you port, they are portable 
scripts. 
>  You port the interpreter, which in autoconf's case is the shell, 
> textutils, and fileutils.  In automake's case it's perl.  If you 
are 
> on the user side of the equation, you don't need autoconf, 
automake, 
> or perl - only their output.

I'm not only on the user side, which may be why I have a different 
opinion then you about portability.  Again, though, this argument is 
pointless, since even perl is something that I can't install.
 
> > 10% of cygwin is still a LOT more then I can install.
> 
> Again, you're confused about needing Cygwin.  Cygwin is at the core 
> cygwin1.dll, a UNIX portability layer that tries to make Windows 
> feel like UNIX to applications built on it (for example, it 
> translates paths used in open calls, treats things like "/dev/null" 
> as special files, etc.).  Mingw32 is a layer that allows you to 
link 
> POSIXish programs against the native Windows libraries and have 
them 
> function like native Windows programs (for example, it uses native 
> Windows paths in open calls).  Cygwin is an emulation layer and 
> Mingw32 is a porting layer.

Sorry, I'm not confused.  I know what cygwin and mingw are... I use 
both of them (though I can only use cygwin at home and avoid it at 
all costs).
 
> As for the size + installation, you won't need to install these 
> programs if you do the bootstrapping method with them like you do 
> now with jam.  The executable size isn't that different, 500k vs. 
> around 2 megs, and unlike jam, most people won't need them anyway 
as 
> they came with their operating system.

The sizes are for the executables.  Translated to source size that's 
quite a bit, and definately too much to bootstrap.  And that's only 
*if* they exist.  Then there's the issue of having to run under a 
shell that's foreign to the system.

> My reccomendations:
> 
> - Really consider using autotools.  I think a lot of the decision 
> not to use autotools was based on a misunderstanding as to what 
they 
> require and what they can do.

You think wrong.
 
> - Add the build support stuff for autotools outside of the main 
> boost tree so those that don't need it, don't have to get it.  
You'd 
> have "boost" and "boost-support", two separate packages.  If you 
> were on a Windows system and didn't already have the needed 
support, 
> you'd first do something like "cd boost-support; ./configure 
> --prefix=c:\tmp\boost-support; make; make install" set your path to 
> include c:\tmp\boost-support, then build boost "cd ../boost; 
> ./configure; make; make install".  Very standard, very clean, and 
> will support any future needs boost has.  You could wrap it into 
one 
> command if you wanted.  Since boost reaches a lot of Windows users, 
> maybe they'd pick up good habits after seeing such a system in use 
> and start using configuration management in their own projects.

There's some effort being done to do this.  However, there's problems 
with this.  Boost developers can't be expected to have to write and 
maintain the "boost-support" scripts, and if they don't you can 
expect several bumps in the road where the source gets out of synch 
with the build scripts.  Other than this, if you want a "boost-
support" then volunteer to do it.
 
Bill Kempf