Subject: Re: [Boost-build] The future of B2?
From: Klemens Morgenstern (klemens.morgenstern_at_[hidden])
Date: 2016-10-04 12:30:21
Am 04.10.2016 um 00:56 schrieb Rene Rivera:
> On Tue, Sep 27, 2016 at 4:31 PM, Klemens Morgenstern
> <klemens.morgenstern_at_[hidden] <mailto:klemens.morgenstern_at_[hidden]>> wrote:
> I do agree. I use boost.build for my own projects, because the
> concepts are great and it's afaik the only build system that can do
> all that, i.e. meta-targets etc. Waht I would love to see is a b3, a
> solid build system-engine entirely plugin-based (be it in C++,
> Python or Chaiscript) completely written in C++ (we have
> boost.python, boost.dll and hopefully boost.process soon) which then
> is not limited to C/C++, but may be extended to support something
> like java via plugin.
> As has been now mentioned a few times.. I think there's agreement in
> that the current b2 implementation is curtailing further progress, in
> most respects IMO. But one thing we will have to wrestle with is
> "external" dependencies. One of the virtues I see of the current
> implementation is its independence. Hence we will have to think hard
> about what libraries we could use. Because I don't fancy bringing in
> large portions of Boost is a good idea for the plain reason that the
> circular dependency makes testing much harder. But that a a topic for
> further discussion :-)
I wouldn't consider that a real problem. If we take boost libraries at
point X in time and build bjam3 from it, and then move boost to version
Y with bjam3, we always would have the prior bjam3 version to test with.
Not nice, but possible.
The only thing it would complicate is bootstraping, but I think that
would be manageable.
> I think if enough boost people are interested in that and are
> willing to spend there time on this, this could succeed. I've got a
> few ideas for that, so if enough people are interested (especially
> those maintaining b2) we could maybe come up with a solid concept.
> Is there any interest in pursuing that? I really don't think CMake
> is the solution for this problem.
> I think there is certainly interest, as is demonstrated by the responses
> here. Which was the main point of this thread. Which now means in
> deciding how to move forward. And to fast forward to you other
> response.. I suspect we can join forces to accelerate this effort.
Ok, then I'll start sharing my thoughts, though they are not completely
worked out yet. If they were, I'd already be working on the system.
I hope it's not too long, but I really think we could build a great
system here, that goes beyond C++.
So what I would love to see is a fast build system purely using C++ and
maybe Python (or ChaiScript) for Plugins. The idea would be to work in
the requirement and meta-targets in at the lowest level. I'm not sure
how to do that exactly, but something like that maybe:
I define feature X (e.g. 'define') to be incompatible for compile;
thereby it will automatically split the generation at the right
location. I.e. :
feature define : incompatible-for c++.compile c.compile ;
This makes a lot of sense for things like linking. If I define the link
feature to be incompatible for linking, but not for compiling, I would
not need to build all the objects twice as I think boost build currently
I would also handle the toolset differently then it now is; currently
it's handled as a feature (which it is) but the default doesn't work.
For example, if I added a java rule, and wrote this
jar foo : bar.java ;
boost.build would try to build that with toolset gcc (which is my
default). The way I'd love to see that handled is that the engine looks
for a matching toolset, which has a generator java->jar and would thus
either select mono or jdk.
(3) Better handling of by-products, compounds & sources
Let's say I build a binary with the coverage information with gcc:
exe foo : bar.cpp : <cxxflags>--coverage <linkflags>--coverage ;
and now I want to access the gcda file. I should be able to do just
reference the target like this ('::' because unlike '.' it's invalid in
alias cov-data : foo::coverage ;
The same would be true I declared a library without defining whether
it's static of shared, by writing this:
lib foo : bar.cpp ;
alias shared_foo : foo::shared_lib ;
As for sources, if I could obtain them from the file would be awesome,
i.e. the direct targets:
alias objects : foo::sources ; #gives me all the .o files.
(4) More generic targets
What I had to do is put the current git-hash into my software, so I ran
a script. I built a script for that, but I would like to be able to add
something like that to the build system.
So the thing I'd do (if I had a git-module) is to create a string-target
(i.e. the hash) which has the date of the last commit (instead of the
date of the last file change):
git-hash my_hash ;
my-generator-script version.cpp : my_hash ;
Another more generic rule would be abstract-targets like runnable. Now
if I have a binary, it would obviously be runnable, but things like *.py
or *.jar are - in general - also. So if I would have rules for that,
this would be awesome, and the generated target should be more generic.
run foo : foo.py : <stdin>bar.txt <stdout>null <stderr>pipe ;
foo::stderr ; #gives me the error log
foo::exit_code ; # gives the return code.
That could then easily be integrated into a few test rules.
(5) Post-build scanners
Afaik boost.build provides scanners for things like include files etc.
to generate dependencies. I'd like to have post-build scanners, which
could be used to check additional generated files. E.g. when building
java you have additional .class files for each nested class. It would be
nice if they could be also collected, so I'd have them not only as
dependency but as part of the target, e.g. when I'd want to build a jar.
This would also require to declare "incomplete targets", i.e. a target
where I don't know exactly what is the result.
(6) Deferred config
When I add a project-specific toolset (e.g. java or mono, as I have) I'd
like to declare their location in the user-config.jam since it's a local
setting, but I cannot invoke "using" since the module isn't defined yet.
So I'd propose that you can write something like "using"
without it trying to invoke the init yet; on import all the config get's
passed to the module.
using java : /somehwere/on/my/machine/java ;
using java : /somehwere/else/on/my/machine/java ; #different version,
determined by the module.
import java ; # now invoke java.init twice
(7) Target output
That could probably be the easiest concept: the build-system should be
able to output all targets with their config to the console, for example
as json. This would allow an IDE to see all targets and provide the
(8) Plugins as Target
If we have a plugin system and we want to allow project-specific
plugins, we'd need to provide a way to build an load them.
import foo.py ; #obvious, python script, just load it
import bar.cpp ; #must be compiled
That would cause the build system to check if the required plugin is
built correctly, if not it would build it and then start over.
II. Syntax & Semantics
(1) Seperate declaration & extensions
For simplicity reasons I would seperate the target declarations and
possible functions (or currently rules) in seperate files. This would
simplify the whole module system and keep the files clean. You'd
basically have a language like make for the declarations and whatever we
choose for extensions.
(2) Feature Semantics:
I'd propose four types (boost.build knows 3)
- requirement : when using this target it ought to be X, elsewise it's
invalid, basicall like a delete
- definition : this target is for X, if not X use another target.
- default-build : obvious
- usage-requirement : requirement passed on to a target depending on
this, not necessarily used by self
lib foo : bar ; #1, declare without anything
lib foo : bar : require <debug-symbols>on : define <link>static ; #2,
What this does is: if I use foo in a target, that requires link=static
it will choose the latter; if I use it in a target which requires
debug-symbols it would though the first one. If I use it in a target
requiring link=static and debug-symbols=off I'd get an error.
(3) Declaration Syntax
I would propose a clearer syntax:
lib foo : bar.cpp : <requirement>42 : <default>stauff : <usage>thingy ;
lib foo : bar.cpp : require requirement=42 with-default default=stuff
for usage=thingy with other-thingy=other-stuff ;
By with = definition, require = requirement, for = usage-requirement,
with-default = default
This feature syntax would be rather clear, and would also allow things
like `define+=OTHER_DEFINE` or boolean values to just be passed.
Maybe we could also add automatic type deduction:
libfoo.a : bar.cpp ;
(4) Function / Rule Syntax
I would completely seperate the syntax of rules (functions declaring
targets) and functions, so it is syntacitally obvious what it is.
exe foo : bar.cpp [ lib thingy : stuff.cpp ] glob(../src) ;
Boost-Build list run by bdawes at acm.org, david.abrahams at rcn.com, gregod at cs.rpi.edu, cpdaniel at pacbell.net, john at johnmaddock.co.uk