Re: [Boost-build] The future of B2?

Subject: Re: [Boost-build] The future of B2?
From: Dominique Devienne (ddevienne_at_[hidden])
Date: 2016-10-05 04:19:57

On Tue, Oct 4, 2016 at 6:30 PM, Klemens Morgenstern <
klemens.morgenstern_at_[hidden]> wrote:

> Am 04.10.2016 um 00:56 schrieb Rene Rivera:
>
>> [...] I suspect we can join forces to accelerate this effort.
>>
>
> Ok, then I'll start sharing my thoughts, though they are not completely
> worked out yet. If they were, I'd already be working on the system.
>
> jar foo : bar.java ;
>
> boost.build would try to build that with toolset gcc (which is my
> default). The way I'd love to see that handled is that the engine looks for
> a matching toolset, which has a generator java->jar and would thus either
> select mono or jdk.
>

FWIW, this is how CppTasks [1] (Ant extension to build C/C++ code) worked,
if I follow you correctly.

You could have several <compiler> tags [2] in your <cc>, and they'd "bid"
against each other
to decide which one was the most appropriate to build the sources, based on
extension,
platforms, etc...

[1] http://ant-contrib.sourceforge.net/cpptasks/
[2] http://ant-contrib.sourceforge.net/cpptasks/antdocs/CompilerDef.html

> (5) Post-build scanners
>
> Afaik boost.build provides scanners for things like include files etc. to
> generate dependencies. I'd like to have post-build scanners, which could be
> used to check additional generated files. E.g. when building java you have
> additional .class files for each nested class. It would be nice if they
> could be also collected, so I'd have them not only as dependency but as
> part of the target, e.g. when I'd want to build a jar.
>
> This would also require to declare "incomplete targets", i.e. a target
> where I don't know exactly what is the result.
>

This was actually a very important aspect of my Java builds in the past,
for which I had to write special Ant extensions.

We were using Java XML Data Binding [3], which generates Java code from XML
schemas. But unlike a Yacc or Lex,
you didn't control the output files, which depended on the schemas of
course, and you also couldn't introspect the
sources to figure that out easily w/o basically almost writing the tool
itself. So it had to be figured out a-posteriori,
and recorded into a "cache" (or build database) for accurate incremental
builds. Building in general is not easy,
but accurate incremental builds are even harder, especially in situations
like this. Some tools take the route of
monitoring file accesses of forked commands (or "native" commands too if
available), but it's not easy to figure
out what is part of the tool, and what is true input/output, and it's often
is not very portable across OS's.

[3] http://www.oracle.com/technetwork/articles/javase/index-140168.html

> (7) Target output
>
> That could probably be the easiest concept: the build-system should be
> able to output all targets with their config to the console, for example as
> json. This would allow an IDE to see all targets and provide the
> corresponding settings.
>

I would go further than that. A build system IMHO should have two stages:

1) A high level first stage, driven from a high level, declarative, human
friendly syntax,
and input files, the whole lot processed by a smart engine to derive all
the minutia of the build itself.
2) A lower level second stage, driven solely by a representation of the
"minutia" in some format,
which deals with all the forking, parallelizing, progress reporting, etc...

A good example of stage 2 would be Ninja [4], although I'm not fond of its
choice of "minutia" representation.
But what Ninja gets right IMHO, is this separation of concern between the
high level, and the low level.

[4] https://ninja-build.org/

Boost.Build seems to be a strong candidate for the first stage (can't say
myself, as most have surmised already),
but how well does it fare as a second stage?

Now for the build "minutia" format, I'd strongly advocate for SQLite.
Because it's small, fast, smart, fantastic IMHO.
It would force to model properly the build state in relational form, and
allow any tool to extract information about
the build with just a few queries. No JSON parsing; incrementally
updatable; compact since binary.

With CTEs now, you can even do hierarchical queries on your dependencies
for example,
if they are not only available in "flattened" form. Builds are graphs (DAGs
hopefully), and
SQLite deals with them great now (those capabilities are not necessarily
used by tools
themselves, but they allow humans like us to introspect/examine/study the
build in
arbitrary ways, and not only the narrow ways the tool hardcodes via e.g.
"dump targets as JSON").

The SQLite intermediate form basically becomes akin to a "protocol" or
"standard",
and perhaps allow swapping between 1st and 2nd stage tools for example.
A Go-based 2nd stage tool as a single statically linked executable with
strong
(and easy) concurrency would be a good Ninja competitor for example :).
But it also allows the 1st stages to use competing syntaxes too.

But the SQLite form would also allow the kind of Tapestry-like "tracing" I
mentioned
previously, but unlike the tracing being just a bunch of output to
sort/sift through, it's all
optionally recorded in a *persistent* manner, in additional (auxiliary)
tables of the relational model,
to be sliced and diced at leisure using SQL. The 1st stage can record/log
in that form all the
transformation and processing it performs, where it loads stuff from,
etc... but that would be
recorded in a "structured" manner that can be queried for. Now that's
powerful!

And having the "digested" build state in relational form (as SQLite DB(s))
also opens
up new tools to for example display information about the build as web
pages, without
those tools needing to understand or parse the human readable 1st stage
syntax, or
(re)implement the Build Engine. All the necessary information is there
already in SQLite,
down to the command line to execute for build a single TU, by clicking on
that web page.

> II. Syntax & Semantics
>

Not getting into that part myself.

I've already written too much above on the "foundations" of a build system
most likely...
Sorry about that. Even though "I don't do builds anymore", it's still an
area of interest to me.
So I'm just trying to chime in to express my views on this matter based on
my experiences,
in the hope of being heard and those ideas considered by those motivated
enough to actually
do something along those lines. Cheers, --DD