Re: [Boost-build] Upgrading the lexer

Subject: Re: [Boost-build] Upgrading the lexer
From: Fabien Chêne (fabien.chene_at_[hidden])
Date: 2018-02-09 13:07:45

Hi Steven,

As the whitespace handling in the bjam language is AFAICT a major
source of denigration, this looks like a tremendous improvement.
Your transition steps look reasonnable. Would you mind advertising
under http://www.boost.org/build/index.html when the scanner-upgrade
branch is merged ?
Preferably in Latest release, but sadly it seems to be unmaintained.

Incidentaly, while talking about major sources of denigration, and
while you are at the lexer, I routinely hear people complaining about
the way arguments are passed to rules.
You have to know by heart that 'requirements' is the third arguments
of the 'lib' rules, and that 'usage-requirements' is the fifth, etc.
However, 'project' has its own way to receive named arguments.
Moreover, Bazel, a new build system from Google that really looks like
Bjam, uses named arguments for their equivalent 'lib', 'exe' rules. I
have seen this perceived as a real advantage.

I was wondering if it were possible to extend the bjam grammar so that:

rule lib ( name : sources * : requirements * )

can be called either (possibly using named arguments coming from the
rule definition):
lib foo : a.C b.C : requirements <define>XXX ;
lib foo : sources a.C b.C : requirements <define>XXX ;
lib foo : requirements <define>XXX : sources a.C b.C ;
lib sources a.C b.C : name foo : requirements <define>XXX ;

What do you think ?

Cheers,

--
Fabien
2018-01-13 23:57 GMT+01:00 Steven Watanabe via Boost-build
<boost-build_at_[hidden]>:
> AMDG
>
> I've implemented a new lexer that handles whitespace
> more intelligently.
> See https://github.com/boostorg/build/tree/scanner-upgrade
>
> Example:
> import testing;
> rule mytest(sources*:requirements*)
> {
>   sources+=[glob x.cpp];
>   requirements+=<link>shared:<define>YY;
>   run $(sources):::$(requirements);
> }
> mytest test.cpp;
>
> Details:
>
> The following symbols are always their own tokens
> when not quoted or escaped:
> '{', '}', ';'
>
> These symbols are independent tokens in contexts where
> the grammar allows them:
> '<', '>', '>=', '<=', '=', '[', ']', '*', '+', '?', '+=', '?=', ':'
>
> Spaces will not break tokens inside variables expansion
> like $(x:J= ).  This is not a breaking change because it
> currently causes a hard error.
>
> In order to reduce the amount of breakage I've
> also added the following special rules:
> - A ':' is not a keyword when it appears in a token
>   which appears to be either a conditional property
>   like <link>shared:<define>X_DLL or a windows absolute
>   path like C:\\Users
> - A '>' is not a keyword if it closes a matching '<',
>   to allow uses like:
>   if <link>shared in $(properties)
>
> The majority of issues appear in regular expressions
> which must be quoted in most cases:
> WRONG: [ MATCH ([.]) : $(x) ]
> RIGHT: [ MATCH "([.])" : $(x) ]
> I don't want to work around this, because it's too
> ambiguous and unlike conditional properties it
> appears relatively rarely in Jamfiles.
>
> This is a major breaking change, so I'm planning to
> split into three steps:
>
> Step 1. Issue a warning for all tokens that will be handled differently.
> Step 2. Turn the warning into an error.
> Step 3. Enable the new lexer.
>
> The scanner-upgrade branch is currently set to step 1.
>
> Thoughts?
>
> In Christ,
> Steven Watanabe
> _______________________________________________
> Unsubscribe & other changes: https://lists.boost.org/mailman/listinfo.cgi/boost-build