On Tue, Oct 4, 2016 at 6:30 PM, Klemens Morgenstern <klemens.morgenstern@gmx.net> wrote:

Am 04.10.2016 um 00:56 schrieb Rene Rivera:

[...] I suspect we can join forces to accelerate this effort.

Ok, then I'll start sharing my thoughts, though they are not completely worked out yet. If they were, I'd already be working on the system.

jar foo : bar.java ;

boost.build would try to build that with toolset gcc (which is my default). The way I'd love to see that handled is that the engine looks for a matching toolset, which has a generator java->jar and would thus either select mono or jdk.

FWIW, this is how CppTasks [1] (Ant extension to build C/C++ code) worked, if I follow you correctly.

You could have several <compiler> tags [2] in your <cc>, and they'd "bid" against each other

to decide which one was the most appropriate to build the sources, based on extension,

platforms, etc...

[1] http://ant-contrib.sourceforge.net/cpptasks/

[2] http://ant-contrib.sourceforge.net/cpptasks/antdocs/CompilerDef.html

(5) Post-build scanners

Afaik boost.build provides scanners for things like include files etc. to generate dependencies. I'd like to have post-build scanners, which could be used to check additional generated files. E.g. when building java you have additional .class files for each nested class. It would be nice if they could be also collected, so I'd have them not only as dependency but as part of the target, e.g. when I'd want to build a jar.

This would also require to declare "incomplete targets", i.e. a target where I don't know exactly what is the result.

This was actually a very important aspect of my Java builds in the past, for which I had to write special Ant extensions.

We were using Java XML Data Binding [3], which generates Java code from XML schemas. But unlike a Yacc or Lex,

you didn't control the output files, which depended on the schemas of course, and you also couldn't introspect the

sources to figure that out easily w/o basically almost writing the tool itself. So it had to be figured out a-posteriori,

and recorded into a "cache" (or build database) for accurate incremental builds. Building in general is not easy,

but accurate incremental builds are even harder, especially in situations like this. Some tools take the route of

monitoring file accesses of forked commands (or "native" commands too if available), but it's not easy to figure

out what is part of the tool, and what is true input/output, and it's often is not very portable across OS's.

[3] http://www.oracle.com/technetwork/articles/javase/index-140168.html

(7) Target output

That could probably be the easiest concept: the build-system should be able to output all targets with their config to the console, for example as json. This would allow an IDE to see all targets and provide the corresponding settings.

I would go further than that. A build system IMHO should have two stages:

1) A high level first stage, driven from a high level, declarative, human friendly syntax,

and input files, the whole lot processed by a smart engine to derive all the minutia of the build itself.

2) A lower level second stage, driven solely by a representation of the "minutia" in some format,

which deals with all the forking, parallelizing, progress reporting, etc...

A good example of stage 2 would be Ninja [4], although I'm not fond of its choice of "minutia" representation.

But what Ninja gets right IMHO, is this separation of concern between the high level, and the low level.

[4] https://ninja-build.org/

Boost.Build seems to be a strong candidate for the first stage (can't say myself, as most have surmised already),

but how well does it fare as a second stage?

Now for the build "minutia" format, I'd strongly advocate for SQLite. Because it's small, fast, smart, fantastic IMHO.

It would force to model properly the build state in relational form, and allow any tool to extract information about

the build with just a few queries. No JSON parsing; incrementally updatable; compact since binary.

With CTEs now, you can even do hierarchical queries on your dependencies for example,

if they are not only available in "flattened" form. Builds are graphs (DAGs hopefully), and

SQLite deals with them great now (those capabilities are not necessarily used by tools

themselves, but they allow humans like us to introspect/examine/study the build in

arbitrary ways, and not only the narrow ways the tool hardcodes via e.g. "dump targets as JSON").

The SQLite intermediate form basically becomes akin to a "protocol" or "standard",

and perhaps allow swapping between 1st and 2nd stage tools for example.

A Go-based 2nd stage tool as a single statically linked executable with strong

(and easy) concurrency would be a good Ninja competitor for example :).

But it also allows the 1st stages to use competing syntaxes too.

But the SQLite form would also allow the kind of Tapestry-like "tracing" I mentioned

previously, but unlike the tracing being just a bunch of output to sort/sift through, it's all

optionally recorded in a *persistent* manner, in additional (auxiliary) tables of the relational model,

to be sliced and diced at leisure using SQL. The 1st stage can record/log in that form all the

transformation and processing it performs, where it loads stuff from, etc... but that would be

recorded in a "structured" manner that can be queried for. Now that's powerful!

And having the "digested" build state in relational form (as SQLite DB(s)) also opens

up new tools to for example display information about the build as web pages, without

those tools needing to understand or parse the human readable 1st stage syntax, or

(re)implement the Build Engine. All the necessary information is there already in SQLite,

down to the command line to execute for build a single TU, by clicking on that web page.

II. Syntax & Semantics

Not getting into that part myself.

I've already written too much above on the "foundations" of a build system most likely...

Sorry about that. Even though "I don't do builds anymore", it's still an area of interest to me.

So I'm just trying to chime in to express my views on this matter based on my experiences,

in the hope of being heard and those ideas considered by those motivated enough to actually

do something along those lines. Cheers, --DD