Re: [boost] [wave] pp-number bug?

From: Edward Diener (eddielee_at_[hidden])
Date: 2006-01-21 20:08:24

Paul Mensonides wrote:
>> -----Original Message-----
>> From: boost-bounces_at_[hidden] 
>> [mailto:boost-bounces_at_[hidden]] On Behalf Of Hartmut Kaiser
> 
>>> Who's right? I'd agree with Paul Mensonides, because "1e" 
>> is a valid 
>>> pp token which is to be formed before applying the ## 
>> operator. But my 
>>> understanding is quite limited in this area so I commit 
>> myself to the 
>>> Gurus.
>> Paul is right and Wave is wrong. That's a known issue with 
>> Wave, it stems from the fact that Wave currently works on C++ 
>> tokens, not pp-tokens as mandated by the Standard. This will 
>> be fixed in the future, when I change Wave to operate on 
>> pp-tokens, rescanning these into C++ tokens after 
>> preprocessing.
> 
> Just so everyone else understands the difference...  A "preprocessing token" is
> a more permissive version of a regular underlying language token.  Most
> preprocessing tokens are exactly the same as tokens (such as operators and
> punctuators like ++, etc.).  Some are slightly different.  For example, keywords
> don't exist yet, so they are just identifier preprocessing tokens.  The biggest
> difference of all is the pp-number preprocessing token.  Regular number tokens
> are divided up into various types (e.g. hex, floating-point, etc.), but
> pp-numbers are general enough to encompass all of them at once without
> specifically knowing about any of them.  As an example, the preprocessing token
> 0__a is a valid pp-number.  Eventually, preprocessing tokens get converted into
> regular tokens (as the result of preprocessing when they are passed on to the
> parser and immediately prior to evaluating the controlling expression in an #if
> or #elif).  Some preprocessing tokens cannot be converted to regular tokens
> (such as 0__a), and if that conversion is attempted, it is an error.  Note that
> not all preprocessing tokens are converted--only those that are passed on to the
> parser (etc.).  E.g.
> 
> #define EAT(x)
> 
> EAT( 0.0.0.0.0 )
>      ^^^^^^^^^
> 
> The marked part is a single pp-number preprocessing token, but because it
> doesn't appear in the output of the preprocessor, no attempt should ever be make
> to convert it to a regular token.  The point being, that preprocessing tokens
> that cannot be converted can still be used without it being an error--which is
> probably *why* preprocessing tokens exist as a separate token type.  What this
> means for an implementation is that pp-number preprocessing tokens, unlike the
> numeric tokens in the underlying language, cannot be (exclusively) stored as
> numeric values, they are more like identifiers or string-literals.

I have never understood the definition of a pp-number in the C++ 
standard. I am using the 1998 version and I read:

ppnumber:
          digit
          . digit
          ppnumber digit
          ppnumber nondigit
          ppnumber e sign
          ppnumber E sign
          ppnumber .

Since "ppnumber digit" and "ppnumber nondigit" covers all sequences 
beginning either with a "digit" or a ".digit" followed by anything else 
which is not a whitespace character or a comment, why specify the last 
three items ?