[Boost-build] The future of B2?

Subject: [Boost-build] The future of B2?
From: Rene Rivera (grafikrobot_at_[hidden])
Date: 2016-10-13 10:03:18

On Tue, Oct 4, 2016 at 11:30 AM, Klemens Morgenstern <
klemens.morgenstern_at_[hidden]
<javascript:_e(%7B%7D,'cvml','klemens.morgenstern_at_[hidden]');>> wrote:

> Am 04.10.2016 um 00:56 schrieb Rene Rivera:
>
>> On Tue, Sep 27, 2016 at 4:31 PM, Klemens Morgenstern
>> <klemens.morgenstern_at_[hidden]
>> <javascript:_e(%7B%7D,'cvml','klemens.morgenstern_at_[hidden]');> <mailto:
>> klemens.morgenstern_at_[hidden]
>> <javascript:_e(%7B%7D,'cvml','klemens.morgenstern_at_[hidden]');>>> wrote:
>>
>>     I do agree. I use boost.build for my own projects, because the
>>     concepts are great and it's afaik the only build system that can do
>>     all that, i.e. meta-targets etc. Waht I would love to see is a b3, a
>>     solid build system-engine entirely plugin-based (be it in C++,
>>     Python or Chaiscript) completely written in C++ (we have
>>     boost.python, boost.dll and hopefully boost.process soon) which then
>>     is not limited to C/C++, but may be extended to support something
>>     like java via plugin.
>>
>>
>> As has been now mentioned a few times.. I think there's agreement in
>> that the current b2 implementation is curtailing further progress, in
>> most respects IMO. But one thing we will have to wrestle with is
>> "external" dependencies. One of the virtues I see of the current
>> implementation is its independence. Hence we will have to think hard
>> about what libraries we could use. Because I don't fancy bringing in
>> large portions of Boost is a good idea for the plain reason that the
>> circular dependency makes testing much harder. But that a a topic for
>> further discussion :-)
>>
>
> I wouldn't consider that a real problem. If we take boost libraries at
> point X in time and build bjam3 from it, and then move boost to version Y
> with bjam3, we always would have the prior bjam3 version to test with. Not
> nice, but possible.
> The only thing it would complicate is bootstraping, but I think that would
> be manageable.

Well... It's certainly something to discuss further :-)

I'll see if I can quickly comment on some of this.. But, one, we should
start a new thread for some of this. And two, I'm putting together some
common design notes and such that I'll post a bit later.

    I think if enough boost people are interested in that and are
>>     willing to spend there time on this, this could succeed. I've got a
>>     few ideas for that, so if enough people are interested (especially
>>     those maintaining b2) we could maybe come up with a solid concept.
>>
>>     Is there any interest in pursuing that? I really don't think CMake
>>     is the solution for this problem.
>>
>>
>> I think there is certainly interest, as is demonstrated by the responses
>> here. Which was the main point of this thread. Which now means in
>> deciding how to move forward. And to fast forward to you other
>> response.. I suspect we can join forces to accelerate this effort.
>>
>
> Ok, then I'll start sharing my thoughts, though they are not completely
> worked out yet. If they were, I'd already be working on the system.
>
> I hope it's not too long, but I really think we could build a great system
> here, that goes beyond C++.
>
> I. Engine
>
> So what I would love to see is a fast build system purely using C++ and
> maybe Python (or ChaiScript) for Plugins. The idea would be to work in the
> requirement and meta-targets in at the lowest level. I'm not sure how to do
> that exactly, but something like that maybe:
>

Even though I'm leaning along the same lines.. I'm not going to close any
doors yet as regards to how we implement this yet.

> (2) Toolsets
>
> I would also handle the toolset differently then it now is; currently it's
> handled as a feature (which it is) but the default doesn't work.
> For example, if I added a java rule, and wrote this
>
> jar foo : bar.java ;
>
> boost.build would try to build that with toolset gcc (which is my
> default). The way I'd love to see that handled is that the engine looks for
> a matching toolset, which has a generator java->jar and would thus either
> select mono or jdk.
>

Not sure I would handle it that way, as it's not usually a straight
one-to-one transformation. But definitely some way to work from the
toolsets out is the way to go in this case.

> (3) Better handling of by-products, compounds & sources
>
> Let's say I build a binary with the coverage information with gcc:
>
> exe foo : bar.cpp : <cxxflags>--coverage <linkflags>--coverage ;
>
> and now I want to access the gcda file. I should be able to do just
> reference the target like this ('::' because unlike '.' it's invalid in a
> filename):
>
> alias cov-data : foo::coverage ;
>
> The same would be true I declared a library without defining whether it's
> static of shared, by writing this:
>
> lib foo : bar.cpp ;
> alias shared_foo : foo::shared_lib ;
>
> As for sources, if I could obtain them from the file would be awesome,
> i.e. the direct targets:
>
> alias objects : foo::sources ; #gives me all the .o files.
>

Interesting.. How is that different than the target requirements? I.e.
foo/<feature>x. Or, for your examples foo/<coverage>, foo/<link>shared (an
already existing option), or foo/<sources>.

(4) More generic targets
>
> What I had to do is put the current git-hash into my software, so I ran a
> script. I built a script for that, but I would like to be able to add
> something like that to the build system.
>

We can certainly try and have more basic "building block" types of targets.
But the current "make" does do a fair job of capturing most uses of generic
targets.

So the thing I'd do (if I had a git-module) is to create a string-target
> (i.e. the hash) which has the date of the last commit (instead of the date
> of the last file change):
>
> git-hash my_hash ;
> my-generator-script version.cpp : my_hash ;
>

Not sure what you mean by "string-target". Especially since it sounds like
something ephemeral, i.e. only temporary to the current b2 execution. And
if you want something persistent then we are talking about a file with
whatever content you want in it, i.e. the hash.

Another more generic rule would be abstract-targets like runnable. Now if I
> have a binary, it would obviously be runnable, but things like *.py or
> *.jar are - in general - also. So if I would have rules for that, this
> would be awesome, and the generated target should be more generic. I.e.:
>
> run foo : foo.py : <stdin>bar.txt <stdout>null <stderr>pipe ;
> foo::stderr ; #gives me the error log
> foo::exit_code ; # gives the return code.

> That could then easily be integrated into a few test rules.
>

First.. The test rules at the moment are rather a mess. They where
complicated to begin with in b1. And they got ported as simply as possible
to b2 from fear of not breaking them.

Second.. I've frequently run in to that issue also. And even though it's
currently possible to run Python files. It's done in a rather circumspect
manner. So, yes, some form of run target with optional launcher would be
good to have.

(5) Post-build scanners
>
> Afaik boost.build provides scanners for things like include files etc. to
> generate dependencies. I'd like to have post-build scanners, which could be
> used to check additional generated files. E.g. when building java you have
> additional .class files for each nested class. It would be nice if they
> could be also collected, so I'd have them not only as dependency but as
> part of the target, e.g. when I'd want to build a jar.
>
> This would also require to declare "incomplete targets", i.e. a target
> where I don't know exactly what is the result.
>

I understand.. But this is not an easy feature to implement. As it
essentially means a mutating build graph while one is building. But it
could be done with some restrictions.

(7) Target output
>
> That could probably be the easiest concept: the build-system should be
> able to output all targets with their config to the console, for example as
> json. This would allow an IDE to see all targets and provide the
> corresponding settings.
>

It's not actually "easy" :-) It's easy to describe but not to implement in
a usable manner. This is because of the dynamic nature of b2 target
descriptions. For integration into IDEs I would lean more towards a library
approach. Where the IDE and b2 can be implemented in terms of that library.
Not it could be a regular link library, but it could also be a server like
API. And depending on the IDE, like VS, I think I like the idea of be
reading the IDE project as the basis of the build.

II. Syntax & Semantics
>
> (1) Seperate declaration & extensions
>
> For simplicity reasons I would seperate the target declarations and
> possible functions (or currently rules) in seperate files. This would
> simplify the whole module system and keep the files clean. You'd basically
> have a language like make for the declarations and whatever we choose for
> extensions.
>

That's about what I had in mind also.

(3) Declaration Syntax
>
> I would propose a clearer syntax:
>
> lib foo : bar.cpp : <requirement>42 : <default>stauff : <usage>thingy ;
> lib foo : bar.cpp : require requirement=42 with-default default=stuff for
> usage=thingy with other-thingy=other-stuff ;
>
> By with = definition, require = requirement, for = usage-requirement,
> with-default = default
>
> This feature syntax would be rather clear, and would also allow things
> like `define+=OTHER_DEFINE` or boolean values to just be passed.
>

I think we are going to disagree  a lot on this area.. As I'm not one for
trying to make things structure like natural languages (I particularly
detest SQL for example). This will certainly need an expanded discussion.
But for now all I'd say is that.. We should avoid departing too far from
current syntax as it will cause undue friction to existing users in
transitioning to a new version.

Maybe we could also add automatic type deduction:
>
> libfoo.a : bar.cpp ;
>

Meh.. Not worth the implementation pain.

-- 
-- Rene Rivera
-- Grafik - Don't Assume Anything
-- Robot Dreams - http://robot-dreams.net
-- rrivera/acm.org (msn) - grafikrobot/aim,yahoo,skype,efnet,gmail
-- 
-- Rene Rivera
-- Grafik - Don't Assume Anything
-- Robot Dreams - http://robot-dreams.net
-- rrivera/acm.org (msn) - grafikrobot/aim,yahoo,skype,efnet,gmail