Re: [boost] [preprocessing] Feedback requested on a C99 preprocessor written in pure universal Python

Subject: Re: [boost] [preprocessing] Feedback requested on a C99 preprocessor written in pure universal Python
From: Niall Douglas (s_sourceforge_at_[hidden])
Date: 2017-03-07 22:23:06

>> I am also *extremely* sure that I could not have developed a conforming
>> C preprocessor in Boost.Spirit in just eighty hours of work. 
> 
> I don't think your preprocessor is conforming. 

It is not. Apart from that one sentence above you chose to quote I have
never claimed it was conforming, and on its Readme page at the front of
its github is a list of known non-conformances.

> I had a quick look at it
> today. From the tests I performed I came away with the impression that while
> a large amount of work has been done, it still requires a lot of work in
> order to turn it into a really conforming preprocessor. I just ran the tests
> of the wave test suite and discovered a diverse set of problems around
> reporting of line number information,

#line isn't implemented as I have no personal need for it, as the Readme
says. pcpp generates line directives, and can pass them through, but
does not parse them.

> macro preprocessing problems (where
> things get rescanned either too often or not often enough - depending on the
> context), non-conforming whitespace placement in the generated output
> (things which should get concatenated are not or v.v.),

These were not known. But I am not surprised. Whitespace implementation
in particular was rushed.

> conditional
> expression evaluation problems, universal character encoding problems,
> missing error reporting for various use cases of conditional preprocessing,
> invalid/missing recognition of preprocessing numbers, and probably more.

These are also stated as known not to work in the Readme. The error
handling isn't ideal either, I cut corners assuming the input is a valid
program.

> I'd suggest that you use the existing test suites (there are several
> available) to verify the functionality of your preprocessor before claiming
> for it to be conforming.

It uses a modified edition of the mcpp suite which it passes. And the
tricky examples given in the C11 standard.

I have the Wave test suite in my directory structure but it is not
committed yet as I haven't written the support code for the unit testing
yet. I intend to commit it, but leave the tests disabled on the CI as I
don't intend to fix failures.

>> Correct
>> recursive function macro expansion turned out to be a real head
>> scratcher, I ended up solving it using a token colouring approach.
> 
> Yes, getting this right is non-trival. The current version of the code does
> not get it right, however. Especially the handling of placeholder tokens is
> tricky, just one example:
> 
> #define NIL
> #define A B NIL
> #define B() anything
> A()
> 
> This should generate 'B()' and not 'anything' (as one might think).

Yes that's a bug, and actually one I knew about already. Another is it
currently can't cope with files included that don't terminate with a
newline. I'll fix those, but I don't intend to bring pcpp up to passing
all of the Wave test suite as pcpp works just fine with my code, and
that's all I need it for.

Those wanting better conformance are welcome to send pull requests.

Niall

-- 
ned Productions Limited Consulting
http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/