RE: [boost] Preprocessor question

From: Paul Mensonides (pmenso57_at_[hidden])
Date: 2005-02-17 17:35:31

> -----Original Message-----
> From: boost-bounces_at_[hidden] 
> [mailto:boost-bounces_at_[hidden]] On Behalf Of Arkadiy Vertleyb

> Hi,
> 
> Suppose I have the following macro:
> 
> #define DO(x)\
>   BOOST_PP_CAT(STEP3, BOOST_PP_CAT(STEP2, BOOST_PP_EXPAND(STEP1 x)))
> 
> #define STEP2STEP1
> 
> Now, DO(5) should IMO expand into STEP35.  For some reason, 
> VC71 expands it into STEP3 5.  Where does the space come 
> from?

It's a VC expansion order bug.  (Note that the EXPAND should have no effect on
this code.)  The space comes from the space between 'STEP1' and 'x'.  That space
*should* have no effect, but doesn't because VC sucks.  One problem, easily
demonstrated here, is that VC is operating on text, not tokens.  E.g.

#define AB 123
#define ID(x) x

ID( ID(A)B ) --> 123

I.e. the 'ID(A)B' causes juxtaposition, which VC reinterprets as token merging
(i.e. concatenation without concatenation).  This is flat out totally wrong.  VC
*should* be operating on tokens (specifically, preprocessing tokens).  The
juxtaposition should merely produce two adjacent, but distinct, tokens.  The
compiler is never, under any circumstances, allowed to introduce whitespace
until translation phase 7, where "whitespace characters separating tokens are no
longer significant."  Let me clarify with another example:

#define ID(x) x

ID(+)+

This code *does not* result in the increment operator ("++").  This code *does
not* result in "+ +"--either the textual sense or the token-oriented sense (i.e.
token(+) space token(+)).  This code results in the + token immediately followed
by another + token with no intervening whitespace.  Note that this is *not* a
problem because the compiler should not be retokenizing anything.  Each
preprocessing token resulting from the preprocessor should be directly converted
to a "regular" token (in translation phase 7).  A standalone preprocessor (i.e.
one that outputs the texual result) must insert spaces to prevent errant
retokenization.  However, it can only do this at the very last moment--after all
othe preprocessing tasks are performed (especially after all stringizing is done
and after all <header-name> tokens are formed).

(In case it isn't already abundantly clear, I despise the VC preprocessor.)

The bug in VC that causes your problem isn't directly caused by text-based
operation, but it is related to it.  It is also related to VC not doing
evaluation order correctly.  It is probably doing concatenation at the same time
as scanning for macro expansion, or something similar.

In any case, you're code should work as expected (except that EXPAND has no
effect related to what you are doing).  VC is wrong--not just in this example,
but in general.

> Can one CAT two tokens, and get a space between them?

No.

> Is it a bug in VC71 or is it supposed to be like this?

It is one of many bugs in VC's preprocessor.  That preprocessor is is flat-out
terrible and is close to the worst preprocessor implementation that exists is
current compilers.

> If I use BOOST_PP_SEQ_CAT outside, it does what I expect.  
> Can anybody explain why?

How are you using SEQ_CAT?  I ask because SEQ_CAT does an ordered concatenation
from left-to-right.  A simple translation of the above:

SEQ_CAT((STEP3)(STEP2)(STEP1 5))

Will never result in a single STEP2STEP1 token (which is supposed to expand to
nothing).  Instead, it will result, successively in STEP3 -> STEP3STEP2 ->
STEP3STEP2STEP1 5.

> (the above exersize may not make sence out of context.  I 
> don't think the context is relevant, although I can provide 
> it if necessary)

No, the context isn't necessary, though there may be a better way to do whatever
it is you're trying to do.  The probable reason that SEQ_CAT is working is that
SEQ_CAT is doing more workarounds to force expansion on VC that should have
happened already (possibly, should have happened a long time ago).

Regards,
Paul Mensonides