Boost logo

Boost :

Subject: Re: [boost] [preprocessor] Sequences vs. All Other Data Structures
From: Paul Mensonides (pmenso57_at_[hidden])
Date: 2012-04-26 20:55:16

On 4/25/2012 5:01 AM, Paul Mensonides wrote:

> However, it is also not *nearly* as complex as the core language--and
> therefore not nearly as difficult to do correctly. In fact, in the last
> 24 hours, I implemented a macro expansion algorithm that, AFAICT, works
> flawlessly--even in all the corner cases. (Grain of salt... I haven't
> tested it very well, however, and it doesn't have a lexer or parser
> attached to it, so it is not a "preprocessor"--it just takes a
sequence of
> preprocessing tokens and a set of existing macro definitions, and
does the
> macro replacement therein.)

Implementation of the above attached--probably lots of room for
improvement and optimization. Apologies for the mega function (in
particular--I didn't feel like refactoring) and the (non-portable) shell
color-coding (I wanted to see blue paint). I built it with g++ 4.7.0,
but I believe 4.6+ should also work.

$ g++ -std=c++11 -I $CHAOS_ROOT 1.cpp

If this wasn't just a toy, I'd probably replace the symbol table with a
trie (radix tree) populated during lexical analysis--especially with how
many common prefixes you get in C++. Also, I'd avoid the various string
comparisons and just compare iterators--which would faster and improve
locality. Also, for output to tty (i.e. preprocess only) there needs to
be state machine to judiciously insert whitespace to prevent erroneous
re-tokenization by later tool. Such generated whitespace can no longer
affect the semantics of the program (whereas before, it can, thanks to

As one can see from this code, a recursive call to the macro replacement
scanner only occurs in one spot: when preparing an actual argument for a
formal argument that is used in the replacement "in the open". Aside
from that, the blue paint, and context changes (implemented here via
virtual tokens), this is a classical stream editor. Macros invocations
are found in the stream, replaced (in the stream) by their replacement
lists (without macro replacement), and scanning resumes at the first
token from the replacement list.

The input is:

#define O 0 +X ## Y A
#define A() A() B
#define B() B() A
#define C(x, y) x ## y
#define D(...) D D ## __VA_ARGS__ __VA_ARGS__ #__VA_ARGS__
#define ID(...) __VA_ARGS__
#define P(p, x) p ## x(P

D(D(0, 1))
ID (

Given no lexer/parser, the above is manually put in in the code. The
output is:

0 <space> + XY <space> A ( ) <space> B <newline>
A ( ) <space> B ( ) <space> A ( ) <space> B <newline>
C <newline>
D <space> DD ( 0 , <space> 1 ) <space> D <space> D0 , <space> 1 <space>
0 , <space> 1 <space> "0, 1" <space> "D(0, 1)" <newline>
<space> <tab> 1 <space> <newline>
P ( 1 , 2 ) , 12 ( P ) <newline>

which is correct.

g++ outputs:

0 +XY A() B
A() B() A() B
D DD(0, 1) D D0, 1 0, 1 "0, 1" "D(0, 1)"

which is correct.

cl outputs:

0 +XY A() B
A() B() A() B
D DD D0, 1 0, 1 "0, 1" D D0, 1 0, 1 "0, 1" "D(0, 1)"

where the 4th and 6th are wrong.

wave outputs:

0 +XY A() B
A() B() A() B
D DD(0, 1) D D0, 1 0, 1 "0, 1" "D(0, 1)"
error: improperly terminated macro invocation or replacement-list
terminates in partial macro expansion (not supported yet): missing ')'

Paul Mensonides

Boost list run by bdawes at, gregod at, cpdaniel at, john at