Re: [Boost-build] RFC: Boost.Build Python Prototype

Subject: Re: [Boost-build] RFC: Boost.Build Python Prototype
From: Stefan Seefeld (stefan_at_[hidden])
Date: 2016-11-15 10:45:15

On 15.11.2016 09:25, Vladimir Prus wrote:
>
> Stefan,
>
> On 15-Nov-16 4:02 PM, Stefan Seefeld wrote:
>
>>> - Yes, metatargets are complex parts of Boost.Build, but I think it's
>>>   an essential parts. The fact that you can generate a metatarget with
>>>   different properties gives quite some flexibility - in particular
>>>   Boost users that to build multiple variants of the libraries, and run
>>>   tests in different combination of properties. When you say that it's
>>>   easy to chain some invocations, I think you just gloss over the
>>> issue.
>>
>> Can you demonstrate (by virtue of a use-case) what I'm missing by not
>> supporting multi-variant builds ? What is the difference (in
>> functionality) between `b2 --toolchain=gcc,clang` and `b3 --tool=cc:gcc
>> && b3 --tool=cc:clang` ?
>
> The primary example is not that, it's this:
>
>   unit-test test_static
>     : test.cpp /boost/test//unit_test_framework : <link>static ;
>   unit-test test_shared
>         : test.cpp /boost/test//unit_test_framework : <link>shared ;
>   unit-test test_generic : test_generic.cpp ;
>
> This builds some parts of the project with two different values of
> the 'link' property, while building the rest only once.

Ah, good point. Yes, I agree, often a single build produces both a
shared and a static version of a library, so that needs to be supported.
I will think about how best to represent that in my prototype.

> Also, "b3 --tool=gcc:gcc & b3 --tool=cc:clang" might work if the
> decision what variants to build is done by user. In case of Boost,
> the decision which variants to build is done inside top-level Jamroot,
> using various properties include the toolset. I don't suppose it
> would be good if users were required to use non-portable shell scripting
> for that.

I don't understand the point you are making. The question is not what
syntax to use to convey the request to build two variants, the question
is where the decision is made.
What I understand from your description is that it's not always clear or
even possible to define multiple build variants to capture the building
of two versions of a library (for example), so it has to be possible to
do both in the same build process.

So something like

  lib1 = library('mylib', sources=[...], link='static')
  lib2 = library('mylib', sources=[...], link='shared')

might work. Of perhaps even

  meta = library('mylib', sources=[...])
  lib1 = meta.expand(link='static')
  lib2 = meta.expand(link='shared')

>
> BTW, may I suggest we don't use "b3", since "b2" stands for
> "Boost.Build", not "Boost.Build V2".

I see ! I shall name it 'b2p2', then :-)

>
>>> - You also seem to not have any notion of portable build properties -
>>>   instead each tool just accepts whatever it pleases. I can't see any
>>>   support for automatically generating build directories based on
>>>   properties. There does not seem to be any support for computing final
>>>   build properties based on requested ones.
>>
>> You are right, that is still missing. Indeed, my putting together the
>> mechanism for communicating properties across targets felt quite ad-hoc,
>> and I would like to consolidate that into something more formal. I'd
>> appreciate your help with that ! :-)
>> And for avoidance of doubt: I'm not at all against establishing build
>> directory naming conventions based on properties. I just don' want to
>> bake that into the tool itself, i.e. I'd rather make that something that
>> can be "plugged in" per project.
>
> It is fine to customize that, but I think build variants that work out
> of box is also key part of Boost.Build appeal. Users should not have to
> configure this manually.

I agree. But from b2p2's point of view everything above (including Boost
itself) is a 'user'. (While I haven't introduced a per-project config
file yet, I have been thinking about it. So that would be a good place
to define things such as build directory layout, default build variant,
etc.)

>
>>> - Looks like the action/target mechanism basically reimplements all
>>>   of Boost.Build generators
>>
>> Yes, that is indeed my goal. Is there anything fundamentally wrong with
>> that approach ? (I'd rather avoid layering the Python front-end on top
>> of a Jam layer, as that leads to overly complex (and thus rigid) code.
>
> Boost.Build generators have nothing to do with Jam code, really. That's
> the code in
> https://github.com/boostorg/build/blob/develop/src/build/generators.py
> that determines the best transformation from sources
> to targets.

Great, this looks like the sort of algorithms my composite targets'
"expand" method would use to generate the inner target dependency graphs,
for example to replace the code in
https://github.com/stefanseefeld/boost.build/blob/proto/src/b3/targets/library.py#L18
(which I threw together merely as a proof-of-concept).

>
>> It is the underlying idea that is important and useful to preserve. So
>> again, can you demonstrate (ideally by virtue of a few use-cases) how my
>> suggested approach isn't able to serve the requirements ? Or is this
>> strictly about terminology, i.e. BBV2 calling it "generators" and me
>> calling it "composite targets" ?
>
> I suppose it's hard to compare 1200 lines of code, plus a bunch of
> actual generators, with a fairly small prototype :-)

Fair enough. But the goal of a prototype (as well as the appeal in
starting over) is to keep things simple, to highlight the outline of an
idea, and not get hung up on details.

>
> Generators are similar to your composite targets, except that they
> come with a selection merchanism. Where you declare 'library' function
> and must use it, Boost.Build declares a generator with target type of
> SHARED_LIB and source type of OBJ and requirements of <toolset>gcc,
> and then maybe another with requirements of <toolset>msvc.
> Then, when generating a 'lib' metatarget, an appropriate generator
> is selected and run, and can in turn use generators to convert its
> sources to OBJ type.
>
> In other words, you have this:
>
>     from b3.targets.library import *
>     hello = library('hello', ['hello.cc'])
>
> and
>
>     def map_src_to_obj(s):
>             return rule(splitext(s)[0]+'.o', s, cxx.compile,
>                         params=dict(link='static'))
>
> Which hardcodes specific 'library' and 'cxx.compile'. In Boost.Build,
> in both places we use a mechanism that uses a generator appropriate
> for the target type and properties, which sounds better than
> hardcoding enough - and has enough of tricky details to make reuse
> worthwhile.

Right, and I fully agree with your approach there. Perhaps we can
outline the mechanism of generators so I can hook a placeholder into my
model ?
Are generators documented or discussed on a conceptual / abstract level
somewhere ?

Again, the code above was not meant to indicate how real composite
targets are to be written. It's meant to demonstrate that generating
targets on-the-fly is indeed possible with my model. I'd love to combine
this with a generic algorithm that can be used to generate these
internal target graphs.

Thanks !

        Stefan

-- 
      ...ich hab' noch einen Koffer in Berlin...