Re: [boost] [preprocessing] Feedback requested on a C99 preprocessor written in pure universal Python

Subject: Re: [boost] [preprocessing] Feedback requested on a C99 preprocessor written in pure universal Python
From: Thomas Heller (thom.heller_at_[hidden])
Date: 2017-03-08 08:35:35

On Mittwoch, 8. März 2017 07:08:00 CET you wrote:
> Am 06.03.2017 11:45 vorm. schrieb "Niall Douglas via Boost" <
> boost_at_[hidden]>:
> 
> Those of you who watch reddit/r/cpp will know I've been working for the
> past month on a pure Python implementation of a C99 conforming
> preprocessor. I am pleased to be able to ask for Boost feedback on a
> fairly high quality implementation:
> 
> https://github.com/ned14/pcpp
> 
> It passes the C11 standard's list of "tricky" preprocessor expansions
> and the mcpp test suite. It has no issue correctly handling the
> preprocessor metaprogramming I've thrown at it from my Boost libraries.
> I'm not going to claim it can handle Boost.Preprocessor or especially
> Boost.VMD and I've tried neither, but pull requests with source code
> fixes adding support for those (with accompanying unit tests) would be
> welcome.
> 
> My main use case for writing this is to assemble my header only Boost
> libraries into a single "drop in and go" file. To that end it has a
> (still buggy) --passthru mode which passes through #define and #undef
> plus any #if logic which uses an undefined macro. I'm still working on
> pass through mode so expect showstopper bugs in that configuration, but
> as a straight highly standards conforming preprocessor it's ready for
> others to use and I welcome any feedback. There are a multitude of use
> cases, everything from running as a conforming preprocessor before
> invoking MSVC to compile right through to parsing, reflection and
> introspection of source code.
> 
> 
> I did something very similar to this wave quite a few years back. We dubbed
> it partial preprocessing. It's still in use in Phoenix, iirc.
> We developed integrations for b2 and later on integrated it into our cmake
> build process for HPX.

Just to explain the process we developed... We found that in prior times, 
before variadic templates, our variadics emulation using Boost.PP iterations 
etc too slow. To mitigate those costs, we decided to use wave to perform those 
preprocessor iterations, thus partial preprocessing. This was perfectly 
feasible to do without sacrificing portability.
Our cmake implementation can be found here:
https://github.com/STEllAR-GROUP/hpx/blob/
18ff9829e732b32d8ff45a56ceaf0c4e3be99033/cmake/HPX_Preprocessing.cmake

What we did is essentially let CMake create a wave.cfg, depending on the 
present system headers, which tells wave to not expand certain macros to 
retain their functionality to hide platform specific irks. The wave.cfg 
template can be found here:
https://github.com/STEllAR-GROUP/hpx/blob/
18ff9829e732b32d8ff45a56ceaf0c4e3be99033/cmake/templates/wave.cfg.in
Usage is here:
https://github.com/STEllAR-GROUP/hpx/blob/
18ff9829e732b32d8ff45a56ceaf0c4e3be99033/preprocess/CMakeLists.txt

With those you can enable specific headers to get partially preprocessed, for 
example here, where only the BOOST_PP_ITERATOR gets expanded, and written to a 
specific file using waves powerful pragmas:
https://github.com/STEllAR-GROUP/hpx/blob/
18ff9829e732b32d8ff45a56ceaf0c4e3be99033/hpx/runtime/components/
memory_block.hpp#L340-L359
The output can be observed here:
https://github.com/STEllAR-GROUP/hpx/tree/
18ff9829e732b32d8ff45a56ceaf0c4e3be99033/hpx/runtime/components/preprocessed

The advantage here is certainly to have a fully compliant C99 preprocessor at 
our disposal. Your usecase might differ slightly, but I am pretty confident 
that with a little change to our process, completely feasible.
The only reason we got rid of wave 2 years ago was that we made variadic 
templates a prerequesite...

HTH,
Thomas