Re: bjam rationale...

From: David Abrahams (dave_at_[hidden])
Date: 2004-12-16 22:39:23

Larry Evans wrote:
> Whew! This must have taken a lot of time and thought. Thanks.
> 
> On 12/16/2004 06:30 AM, David Abrahams wrote:
> > Tompa wrote:
> [snip]
> 
> >>Fine, thank you! But, hmmm... What is it in that list of
> >>requirements that bjam does so much better than gnumake?
> 
> [snip]
> 
> > Q1.0: A developer adding a new library or test program must only
> > have to add simple entries naming the source files to a text file,
> > and not have to know anything about platform specific files. ...
> 
> Any platform specific files can be included with
> 
> include $(PLATFORM).imk

That's not what Q1.0 means. It means that the build specification for a
library or test program must not contain any platform specific information.

> where PLATFORM is set by testing which platform (I assume this
> means operating system in order to decide the suffixes for
> executables and object files and directory name separators,
> etc.) I'm assuming the test for platform can be done by invoking
> a script function via the $(shell *command*), where *command*
> is the actual script command.

No. Shell commands are highly nonportable.

> > A1.0: With GNU Make a developer needs to write the specific flags
> > and options used by all the compilers and operating systems (s)he is
> > targeting into her build actions. With Boost.Build, the developer
> > writes largely platform- independent build specifications.
> 
> include $(PLATFORM).$(COMPILER).imk

The developer shouldn't have to write stuff like that.

> could include the compiler commands for compiler, $(COMPILER),
> and operating system, $(PLATFORM).

That isn't enough. Suppose your library or program needs threading
enabled or needs inlining turned off? You need to build an
infrastructure for translating compiler-independent descriptions of
these options into ones that depend on the compiler. And the
programming language provided by GNU Make is too weak to build the
capabilities of Boost.BUild and yield maintainable program of manageable
size.

> > Q1.5: ...The developer should not have to supply header dependency
> > information.
> >
> > A1.5: Without help from external tools, GNU Make is not capable of
> > scanning for header dependencies, so the user has to use external tools
> > like perl scripts to handle that job.
> 
> True, but maybe that's not a big disadvantage. 

It was a big disadvantage when Boost.Build was started. If people were
resistant to installing GNU Make on their systems, imagine asking them
to install GNU Make and Perl.

> For example, gcc can
> produce this information in a .d file with the -MD compiler option,
> and I believe most compilers have similar flags to produce the same
> information. 

Whatever your definition of "most" is, it isn't enough. Plenty of
important compilers don't include that support.

> So one might say bjam reinvents the wheel with this
> feature. OTOH, I understand that the dependency information provided
> by bjam is always up-to-date since it does the analysis each time.

It does not, in BBv1 we use a header cache similar to the one created by
gcc.

> OTOOH, this extra scanning requires, I assume, a lot of time.

It does not. Vladimir removed the header cacheing for BBv2 and found no
measurable slowdown. That is not where the bottlenecks are in the
Boost.Build process.

> > Q2. There should be a very high likelihood of builds succeeding on
> > all platforms if a build succeeds on any platform. In other words, a
> > developer must not be required to have access to many platforms or
> > compilers to ensure correct builds
> 
> >
> > A2. See A1.0. Yes, it is possible to write "cross-platform" and
> > "cross-compiler" Makefiles. I have seen such systems, though, and
> > they: 1. have always been limited to a small number of platforms and
> > compilers and 2. have been ugly an unmaintainable hacks because of
> > the expressive limitations of the make language.
> 
> The following part of a Makefile:
> 
> <---------
> HOW=gcc
> #HOW=icc

This is already unacceptable. A user wanting to initiate a build should
not be required to edit a Makefile.

> COMPILER.gcc=/usr/local/gcc-3.4.1/bin/g++
> COMPILER.icc=icc
> COMPILE.gcc=$(COMPILER.gcc) -c -Wall -ftemplate-depth-100 -O0 -fno-inline
> COMPILE.icc=$(COMPILER.icc) -c -xc++ -O0 -Ob0

And this is too is unacceptable. The person writing a library should
not be required to understand the flags for all the compilers that are
targeted.

> LINK.gcc = $(COMPILER.gcc)
> LINK.icc = $(COMPILER.icc)
> ....
> $(SRC).$(HOW).o: $(SRC).cpp
> $(COMPILE.$(HOW)) $(INCS) $(SRC).cpp -o $@
> 
> >---------
> 
> Allows easy selection of the compiler. A similar scheme could be used
> to make the command specific to the platform. The syntax is obviously
> not as clear as bjam's Jamfile's, but I think it still is
> maintainable. 

It doesn't satsify the requirements. You can claim the requirements are
unimportant all you like, but they were developed through extensive
discussion on the Boost list and I won't abandon them lightly.

> After all, adding a new compiler just means adding a
> new instance of the COMPILE.* macros. The user making the change
> would, of course, have to know the relevant commands for the specific
> compiler, but this is the same for bjam. 

The difference is that the person adding the new compiler doesn't
intrude on any makefiles.

> And with bjam, when I wanted
> to add como to the set of compilers, I had to search through several
> source files before I figured I needed to modify ~/user-config.jam and
> needed to add tools/como.jam or something like that.

That's a documentation problem (which I hope will be fixed soon), not an
architecture problem.

> { BTW, I think the $(COMPILE.$(HOW)) usage is gnumake specific.}
> 
> In addition, the $(SRC).$(HOW).o in the above Makefile part could be
> extended to emulate the way bjam selects the output directory as
> follows:
> 
> $(BUILD)/$(WHAT)/$(HOW)/$(SRC).o
> 
> where WHAT could be the location w.r.t. some root source directory and
> BUILD would be the build-dir, in bjam.v2 terms.

That's an exceedingly simplistic view of what Boost.Build is doing. And
there's no such thing as bjam.v2.

> > A3. In other words, it should be possible to plug in support for a
> > new operating system or compiler without modifying the project
> > descriptions (makefiles/Jamfiles, whatever). With GNU make that
> > isn't supported natively and is difficult to implement.
> 
> I think it could be done with the:
> 
> include $(PLATFORM).$(COMPILER).imk
> 
> scheme mentioned earlier. And BTW again, I think this `include`
> command is gnumake specific too.

As I mentioned earlier, Makefile authors should not be required to utter
incantations like that. One of the problems with the architecture of
all Makes is that the Makefile always gets control before anything else.
If there's a build system infrastructure, you have to invoke it from
the Makefile, and the overall flow of control is ultimately managed by
the user's Makefile. As a result, the user has to be concerned with
low-level details that she shouldn't have to worry about. Boost.Build
loads the build system infrastructure and then reads Jamfiles under
control of the build system.

> > Q6. It should be possible to build multiple variants (e.g.
> > debug/release) of a single target.
> 
> This could be done by including a file located in a directory whose
> path is composed of the variants. For example:
> 
> HOW=$(COMPILER)/$(VARIANT)
> HOW.path=./how_variants/$(HOW)
> include $(HOW.path)/how.imk
> 
> where:
> 
> VARIANT = "debug" or "release" or whatever the user chooses as long
> as he provides the corresponding how.imk file to be included.

Again, you can do it, but it's ugly, and it intrudes on Makefiles with
low-level details.

> > Q7. It should be possible to build multiple variants of multiple targets
> > with multiple compilers from a single build command.
> 
> > A7. Same answer, multiplied by 10.
> 
> I don't think make can do this from a single command, 

Sure it can. But it's awful. You can make Make do almost all of this,
but its poor expressivity makes it painful and the result is
hard-to-maintain. I don't think the bjam language is so wonderful
either (one reason we're talking about Python), but it's a whole heck of
a lot better than the GNU Make language for this sort of thing.

> unless the
> command could be parsed to detect multiple variants/targets and then
> make invoked with each combination. This would have to be done by
> some script file, AFAICT.
> 
> [snip]
> 
> > Q9. Support for dynamic and static linking should be included.
> 
> Again, this could be done by encoding the type of link in the VARIANT
> macro mentioned after Q6 above. However, instead of just release or
> debug, there would be macros,
> 
> LINK-TYPE # either "shared" or "static"
> DEV-PHASE # either "debug" or "release"

Simple-minded schemes like this don't support the ability to link some
libraries statically and others shared.

> > Q13. It should be possible to build targets into a directory
> > unrelated to the source directories (they may be read-only)
> 
> > A14. I *think* GNU make can handle this easily enough if properly
> > engineered to do so. But while we're in the neighborhood, Google
> > for "recursive make considered harmful"
> 
> My above response containing $(BUILD) suggests how this could be done
> without recursive makes.

It all can be done, or nearly all (except for the header scanning), but
in this case it's anti-idiomatic. Anyway, I don't see how BUILD helps
with the recursive make issue -- that concerns how to construct projects
software that's described across multiple makefiles.

-- 
Dave Abrahams
Boost Consulting
http://www.boost-consulting.com