On Tue, Oct 4, 2016 at 11:30 AM, Klemens Morgenstern <klemens.morgenstern@gmx.net> wrote:
Am 04.10.2016 um 00:56 schrieb Rene Rivera:
On Tue, Sep 27, 2016 at 4:31 PM, Klemens Morgenstern
<klemens.morgenstern@gmx.net <mailto:klemens.morgenstern@gmx.net>> wrote:

    I do agree. I use boost.build for my own projects, because the
    concepts are great and it's afaik the only build system that can do
    all that, i.e. meta-targets etc. Waht I would love to see is a b3, a
    solid build system-engine entirely plugin-based (be it in C++,
    Python or Chaiscript) completely written in C++ (we have
    boost.python, boost.dll and hopefully boost.process soon) which then
    is not limited to C/C++, but may be extended to support something
    like java via plugin.

As has been now mentioned a few times.. I think there's agreement in
that the current b2 implementation is curtailing further progress, in
most respects IMO. But one thing we will have to wrestle with is
"external" dependencies. One of the virtues I see of the current
implementation is its independence. Hence we will have to think hard
about what libraries we could use. Because I don't fancy bringing in
large portions of Boost is a good idea for the plain reason that the
circular dependency makes testing much harder. But that a a topic for
further discussion :-)

I wouldn't consider that a real problem. If we take boost libraries at point X in time and build bjam3 from it, and then move boost to version Y with bjam3, we always would have the prior bjam3 version to test with. Not nice, but possible.
The only thing it would complicate is bootstraping, but I think that would be manageable.

Well... It's certainly something to discuss further :-) 

I'll see if I can quickly comment on some of this.. But, one, we should start a new thread for some of this. And two, I'm putting together some common design notes and such that I'll post a bit later.

    I think if enough boost people are interested in that and are
    willing to spend there time on this, this could succeed. I've got a
    few ideas for that, so if enough people are interested (especially
    those maintaining b2) we could maybe come up with a solid concept.

    Is there any interest in pursuing that? I really don't think CMake
    is the solution for this problem.

I think there is certainly interest, as is demonstrated by the responses
here. Which was the main point of this thread. Which now means in
deciding how to move forward. And to fast forward to you other
response.. I suspect we can join forces to accelerate this effort.

Ok, then I'll start sharing my thoughts, though they are not completely worked out yet. If they were, I'd already be working on the system.

I hope it's not too long, but I really think we could build a great system here, that goes beyond C++.

I. Engine

So what I would love to see is a fast build system purely using C++ and maybe Python (or ChaiScript) for Plugins. The idea would be to work in the requirement and meta-targets in at the lowest level. I'm not sure how to do that exactly, but something like that maybe:

Even though I'm leaning along the same lines.. I'm not going to close any doors yet as regards to how we implement this yet.
(2) Toolsets

I would also handle the toolset differently then it now is; currently it's handled as a feature (which it is) but the default doesn't work.
For example, if I added a java rule, and wrote this

jar foo : bar.java ;

boost.build would try to build that with toolset gcc (which is my default). The way I'd love to see that handled is that the engine looks for a matching toolset, which has a generator java->jar and would thus either select mono or jdk.

Not sure I would handle it that way, as it's not usually a straight one-to-one transformation. But definitely some way to work from the toolsets out is the way to go in this case.
(3) Better handling of by-products, compounds & sources

Let's say I build a binary with the coverage information with gcc:

exe foo : bar.cpp : <cxxflags>--coverage <linkflags>--coverage ;

and now I want to access the gcda file. I should be able to do just reference the target like this ('::' because unlike '.' it's invalid in a filename):

alias cov-data : foo::coverage ;

The same would be true I declared a library without defining whether it's static of shared, by writing this:

lib foo : bar.cpp ;
alias shared_foo : foo::shared_lib ;

As for sources, if I could obtain them from the file would be awesome, i.e. the direct targets:

alias objects : foo::sources ; #gives me all the .o files.

Interesting.. How is that different than the target requirements? I.e. foo/<feature>x. Or, for your examples foo/<coverage>, foo/<link>shared (an already existing option), or foo/<sources>.

(4) More generic targets

What I had to do is put the current git-hash into my software, so I ran a script. I built a script for that, but I would like to be able to add something like that to the build system.

We can certainly try and have more basic "building block" types of targets. But the current "make" does do a fair job of capturing most uses of generic targets. 

So the thing I'd do (if I had a git-module) is to create a string-target (i.e. the hash) which has the date of the last commit (instead of the date of the last file change):

git-hash my_hash ;
my-generator-script version.cpp : my_hash ;

Not sure what you mean by "string-target". Especially since it sounds like something ephemeral, i.e. only temporary to the current b2 execution. And if you want something persistent then we are talking about a file with whatever content you want in it, i.e. the hash. 

Another more generic rule would be abstract-targets like runnable. Now if I have a binary, it would obviously be runnable, but things like *.py or *.jar are - in general - also. So if I would have rules for that, this would be awesome, and the generated target should be more generic. I.e.:

run foo : foo.py : <stdin>bar.txt <stdout>null <stderr>pipe ;
foo::stderr ; #gives me the error log
foo::exit_code ; # gives the return code. 

That could then easily be integrated into a few test rules.

First.. The test rules at the moment are rather a mess. They where complicated to begin with in b1. And they got ported as simply as possible to b2 from fear of not breaking them.

Second.. I've frequently run in to that issue also. And even though it's currently possible to run Python files. It's done in a rather circumspect manner. So, yes, some form of run target with optional launcher would be good to have. 

(5) Post-build scanners

Afaik boost.build provides scanners for things like include files etc. to generate dependencies. I'd like to have post-build scanners, which could be used to check additional generated files. E.g. when building java you have additional .class files for each nested class. It would be nice if they could be also collected, so I'd have them not only as dependency but as part of the target, e.g. when I'd want to build a jar.

This would also require to declare "incomplete targets", i.e. a target where I don't know exactly what is the result.

I understand.. But this is not an easy feature to implement. As it essentially means a mutating build graph while one is building. But it could be done with some restrictions. 

(7) Target output

That could probably be the easiest concept: the build-system should be able to output all targets with their config to the console, for example as json. This would allow an IDE to see all targets and provide the corresponding settings.

It's not actually "easy" :-) It's easy to describe but not to implement in a usable manner. This is because of the dynamic nature of b2 target descriptions. For integration into IDEs I would lean more towards a library approach. Where the IDE and b2 can be implemented in terms of that library. Not it could be a regular link library, but it could also be a server like API. And depending on the IDE, like VS, I think I like the idea of be reading the IDE project as the basis of the build. 

II. Syntax & Semantics

(1) Seperate declaration & extensions

For simplicity reasons I would seperate the target declarations and possible functions (or currently rules) in seperate files. This would simplify the whole module system and keep the files clean. You'd basically have a language like make for the declarations and whatever we choose for extensions.

That's about what I had in mind also.

(3) Declaration Syntax

I would propose a clearer syntax:

lib foo : bar.cpp : <requirement>42 : <default>stauff : <usage>thingy ;
lib foo : bar.cpp : require requirement=42 with-default default=stuff for usage=thingy with other-thingy=other-stuff ;

By with = definition, require = requirement, for = usage-requirement, with-default = default

This feature syntax would be rather clear, and would also allow things like `define+=OTHER_DEFINE` or boolean values to just be passed.

I think we are going to disagree  a lot on this area.. As I'm not one for trying to make things structure like natural languages (I particularly detest SQL for example). This will certainly need an expanded discussion. But for now all I'd say is that.. We should avoid departing too far from current syntax as it will cause undue friction to existing users in transitioning to a new version.

Maybe we could also add automatic type deduction:

libfoo.a : bar.cpp ;

Meh.. Not worth the implementation pain. 

-- Rene Rivera
-- Grafik - Don't Assume Anything
-- Robot Dreams - http://robot-dreams.net
-- rrivera/acm.org (msn) - grafikrobot/aim,yahoo,skype,efnet,gmail

-- Rene Rivera
-- Grafik - Don't Assume Anything
-- Robot Dreams - http://robot-dreams.net
-- rrivera/acm.org (msn) - grafikrobot/aim,yahoo,skype,efnet,gmail