Re: [boost] [preprocessing] Feedback requested on a C99 preprocessor written in pure universal Python

Subject: Re: [boost] [preprocessing] Feedback requested on a C99 preprocessor written in pure universal Python
From: Hartmut Kaiser (hartmut.kaiser_at_[hidden])
Date: 2017-03-07 20:36:11

> I discounted Wave as I felt its implementation was not flexible enough
> for this use case of dynamic rewriting. 

Wave has a full command line based preprocessor ready to use (see
https://github.com/boostorg/wave/tree/develop/tool). No work on your end
should have been necessary at all.

> Also an implementation written
> in C or C++ cannot be dynamically driven by a build process like a
> Python implementation can e.g. cmake can write Python script into a file
> and inject that into pcpp.

Sorry, I don't understand what this means. Could you elaborate, please?

> I am also *extremely* sure that I could not have developed a conforming
> C preprocessor in Boost.Spirit in just eighty hours of work. 

I don't think your preprocessor is conforming. I had a quick look at it
today. From the tests I performed I came away with the impression that while
a large amount of work has been done, it still requires a lot of work in
order to turn it into a really conforming preprocessor. I just ran the tests
of the wave test suite and discovered a diverse set of problems around
reporting of line number information, macro preprocessing problems (where
things get rescanned either too often or not often enough - depending on the
context), non-conforming whitespace placement in the generated output
(things which should get concatenated are not or v.v.), conditional
expression evaluation problems, universal character encoding problems,
missing error reporting for various use cases of conditional preprocessing,
invalid/missing recognition of preprocessing numbers, and probably more.

I'd suggest that you use the existing test suites (there are several
available) to verify the functionality of your preprocessor before claiming
for it to be conforming.

> Correct
> recursive function macro expansion turned out to be a real head
> scratcher, I ended up solving it using a token colouring approach.

Yes, getting this right is non-trival. The current version of the code does
not get it right, however. Especially the handling of placeholder tokens is
tricky, just one example:

#define NIL
#define A B NIL
#define B() anything
A()

This should generate 'B()' and not 'anything' (as one might think).

HTH
Regards Hartmut
---------------
http://boost-spirit.com
http://stellar.cct.lsu.edu