Re: [boost] [wave] pp-number bug?

From: Paul Mensonides (pmenso57_at_[hidden])
Date: 2006-01-21 18:57:29

> -----Original Message-----
> From: boost-bounces_at_[hidden] 
> [mailto:boost-bounces_at_[hidden]] On Behalf Of Hartmut Kaiser

> > Who's right? I'd agree with Paul Mensonides, because "1e" 
> is a valid 
> > pp token which is to be formed before applying the ## 
> operator. But my 
> > understanding is quite limited in this area so I commit 
> myself to the 
> > Gurus.
> 
> Paul is right and Wave is wrong. That's a known issue with 
> Wave, it stems from the fact that Wave currently works on C++ 
> tokens, not pp-tokens as mandated by the Standard. This will 
> be fixed in the future, when I change Wave to operate on 
> pp-tokens, rescanning these into C++ tokens after 
> preprocessing.

Just so everyone else understands the difference...  A "preprocessing token" is
a more permissive version of a regular underlying language token.  Most
preprocessing tokens are exactly the same as tokens (such as operators and
punctuators like ++, etc.).  Some are slightly different.  For example, keywords
don't exist yet, so they are just identifier preprocessing tokens.  The biggest
difference of all is the pp-number preprocessing token.  Regular number tokens
are divided up into various types (e.g. hex, floating-point, etc.), but
pp-numbers are general enough to encompass all of them at once without
specifically knowing about any of them.  As an example, the preprocessing token
0__a is a valid pp-number.  Eventually, preprocessing tokens get converted into
regular tokens (as the result of preprocessing when they are passed on to the
parser and immediately prior to evaluating the controlling expression in an #if
or #elif).  Some preprocessing tokens cannot be converted to regular tokens
(such as 0__a), and if that conversion is attempted, it is an error.  Note that
not all preprocessing tokens are converted--only those that are passed on to the
parser (etc.).  E.g.

#define EAT(x)

EAT( 0.0.0.0.0 )
     ^^^^^^^^^

The marked part is a single pp-number preprocessing token, but because it
doesn't appear in the output of the preprocessor, no attempt should ever be make
to convert it to a regular token.  The point being, that preprocessing tokens
that cannot be converted can still be used without it being an error--which is
probably *why* preprocessing tokens exist as a separate token type.  What this
means for an implementation is that pp-number preprocessing tokens, unlike the
numeric tokens in the underlying language, cannot be (exclusively) stored as
numeric values, they are more like identifiers or string-literals.

Regards,
Paul Mensonides