Re: [Boost-build] The future of B2?

Subject: Re: [Boost-build] The future of B2?
From: Klemens Morgenstern (klemens.morgenstern_at_[hidden])
Date: 2016-10-04 12:30:21

Am 04.10.2016 um 00:56 schrieb Rene Rivera:
> On Tue, Sep 27, 2016 at 4:31 PM, Klemens Morgenstern
> <klemens.morgenstern_at_[hidden] <mailto:klemens.morgenstern_at_[hidden]>> wrote:
>
>     I do agree. I use boost.build for my own projects, because the
>     concepts are great and it's afaik the only build system that can do
>     all that, i.e. meta-targets etc. Waht I would love to see is a b3, a
>     solid build system-engine entirely plugin-based (be it in C++,
>     Python or Chaiscript) completely written in C++ (we have
>     boost.python, boost.dll and hopefully boost.process soon) which then
>     is not limited to C/C++, but may be extended to support something
>     like java via plugin.
>
>
> As has been now mentioned a few times.. I think there's agreement in
> that the current b2 implementation is curtailing further progress, in
> most respects IMO. But one thing we will have to wrestle with is
> "external" dependencies. One of the virtues I see of the current
> implementation is its independence. Hence we will have to think hard
> about what libraries we could use. Because I don't fancy bringing in
> large portions of Boost is a good idea for the plain reason that the
> circular dependency makes testing much harder. But that a a topic for
> further discussion :-)

I wouldn't consider that a real problem. If we take boost libraries at 
point X in time and build bjam3 from it, and then move boost to version 
Y with bjam3, we always would have the prior bjam3 version to test with. 
Not nice, but possible.
The only thing it would complicate is bootstraping, but I think that 
would be manageable.

>
>
>     I think if enough boost people are interested in that and are
>     willing to spend there time on this, this could succeed. I've got a
>     few ideas for that, so if enough people are interested (especially
>     those maintaining b2) we could maybe come up with a solid concept.
>
>     Is there any interest in pursuing that? I really don't think CMake
>     is the solution for this problem.
>
>
> I think there is certainly interest, as is demonstrated by the responses
> here. Which was the main point of this thread. Which now means in
> deciding how to move forward. And to fast forward to you other
> response.. I suspect we can join forces to accelerate this effort.

Ok, then I'll start sharing my thoughts, though they are not completely 
worked out yet. If they were, I'd already be working on the system.

I hope it's not too long, but I really think we could build a great 
system here, that goes beyond C++.

I. Engine

So what I would love to see is a fast build system purely using C++ and 
maybe Python (or ChaiScript) for Plugins. The idea would be to work in 
the requirement and meta-targets in at the lowest level. I'm not sure 
how to do that exactly, but something like that maybe:

(1) Features

I define feature X (e.g. 'define') to be incompatible for compile; 
thereby it will automatically split the generation at the right 
location. I.e. :

feature define : incompatible-for c++.compile c.compile ;

This makes a lot of sense for things like linking. If I define the link 
feature to be incompatible for linking, but not for compiling, I would 
not need to build all the objects twice as I think boost build currently 
does.

(2) Toolsets

I would also handle the toolset differently then it now is; currently 
it's handled as a feature (which it is) but the default doesn't work.
For example, if I added a java rule, and wrote this

jar foo : bar.java ;

boost.build would try to build that with toolset gcc (which is my 
default). The way I'd love to see that handled is that the engine looks 
for a matching toolset, which has a generator java->jar and would thus 
either select mono or jdk.

(3) Better handling of by-products, compounds & sources

Let's say I build a binary with the coverage information with gcc:

exe foo : bar.cpp : <cxxflags>--coverage <linkflags>--coverage ;

and now I want to access the gcda file. I should be able to do just 
reference the target like this ('::' because unlike '.' it's invalid in 
a filename):

alias cov-data : foo::coverage ;

The same would be true I declared a library without defining whether 
it's static of shared, by writing this:

lib foo : bar.cpp ;
alias shared_foo : foo::shared_lib ;

As for sources, if I could obtain them from the file would be awesome, 
i.e. the direct targets:

alias objects : foo::sources ; #gives me all the .o files.

(4) More generic targets

What I had to do is put the current git-hash into my software, so I ran 
a script. I built a script for that, but I would like to be able to add 
something like that to the build system.

So the thing I'd do (if I had a git-module) is to create a string-target 
(i.e. the hash) which has the date of the last commit (instead of the 
date of the last file change):

git-hash my_hash ;
my-generator-script version.cpp : my_hash ;

Another more generic rule would be abstract-targets like runnable. Now 
if I have a binary, it would obviously be runnable, but things like *.py 
or *.jar are - in general - also. So if I would have rules for that, 
this would be awesome, and the generated target should be more generic. 
I.e.:

run foo : foo.py : <stdin>bar.txt <stdout>null <stderr>pipe ;
foo::stderr ; #gives me the error log
foo::exit_code ; # gives the return code.

That could then easily be integrated into a few test rules.

(5) Post-build scanners

Afaik boost.build provides scanners for things like include files etc. 
to generate dependencies. I'd like to have post-build scanners, which 
could be used to check additional generated files. E.g. when building 
java you have additional .class files for each nested class. It would be 
nice if they could be also collected, so I'd have them not only as 
dependency but as part of the target, e.g. when I'd want to build a jar.

This would also require to declare "incomplete targets", i.e. a target 
where I don't know exactly what is the result.

(6) Deferred config

When I add a project-specific toolset (e.g. java or mono, as I have) I'd 
like to declare their location in the user-config.jam since it's a local 
setting, but I cannot invoke "using" since the module isn't defined yet. 
So I'd propose that you can write something like "using"
without it trying to invoke the init yet; on import all the config get's 
passed to the module.

I.e.:

#user-config.jam
using java : /somehwere/on/my/machine/java ;
using java : /somehwere/else/on/my/machine/java ; #different version, 
determined by the module.

#Jamroot.jam
import java ; # now invoke java.init twice

(7) Target output

That could probably be the easiest concept: the build-system should be 
able to output all targets with their config to the console, for example 
as json. This would allow an IDE to see all targets and provide the 
corresponding settings.

(8) Plugins as Target

If we have a plugin system and we want to allow project-specific 
plugins, we'd need to provide a way to build an load them.

import foo.py ; #obvious, python script, just load it
import bar.cpp ; #must be compiled

That would cause the build system to check if the required plugin is 
built correctly, if not it would build it and then start over.

II. Syntax & Semantics

(1) Seperate declaration & extensions

For simplicity reasons I would seperate the target declarations and 
possible functions (or currently rules) in seperate files. This would 
simplify the whole module system and keep the files clean. You'd 
basically have a language like make for the declarations and whatever we 
choose for extensions.

(2) Feature Semantics:

I'd propose four types (boost.build knows 3)

- requirement : when using this target it ought to be X, elsewise it's 
invalid, basicall like a delete
- definition : this target is for X, if not X use another target.
- default-build : obvious
- usage-requirement : requirement passed on to a target depending on 
this, not necessarily used by self

lib foo : bar ; #1, declare without anything
lib foo : bar : require <debug-symbols>on : define <link>static ; #2,

What this does is: if I use foo in a target, that requires link=static 
it will choose the latter; if I use it in a target which requires 
debug-symbols it would though the first one. If I use it in a target 
requiring link=static and debug-symbols=off I'd get an error.

(3) Declaration Syntax

I would propose a clearer syntax:

lib foo : bar.cpp : <requirement>42 : <default>stauff : <usage>thingy ;
lib foo : bar.cpp : require requirement=42 with-default default=stuff 
for usage=thingy with other-thingy=other-stuff ;

By with = definition, require = requirement, for = usage-requirement, 
with-default = default

This feature syntax would be rather clear, and would also allow things 
like `define+=OTHER_DEFINE` or boolean values to just be passed.

Maybe we could also add automatic type deduction:

libfoo.a : bar.cpp ;

(4) Function / Rule Syntax

I would completely seperate the syntax of rules (functions declaring 
targets) and functions, so it is syntacitally obvious what it is.

That is:

exe foo : bar.cpp [ lib thingy : stuff.cpp ] glob(../src) ;