Boost logo

Boost :

From: Paul Mensonides (pmenso57_at_[hidden])
Date: 2006-01-21 18:57:29

> -----Original Message-----
> From: boost-bounces_at_[hidden]
> [mailto:boost-bounces_at_[hidden]] On Behalf Of Hartmut Kaiser

> > Who's right? I'd agree with Paul Mensonides, because "1e"
> is a valid
> > pp token which is to be formed before applying the ##
> operator. But my
> > understanding is quite limited in this area so I commit
> myself to the
> > Gurus.
> Paul is right and Wave is wrong. That's a known issue with
> Wave, it stems from the fact that Wave currently works on C++
> tokens, not pp-tokens as mandated by the Standard. This will
> be fixed in the future, when I change Wave to operate on
> pp-tokens, rescanning these into C++ tokens after
> preprocessing.

Just so everyone else understands the difference... A "preprocessing token" is
a more permissive version of a regular underlying language token. Most
preprocessing tokens are exactly the same as tokens (such as operators and
punctuators like ++, etc.). Some are slightly different. For example, keywords
don't exist yet, so they are just identifier preprocessing tokens. The biggest
difference of all is the pp-number preprocessing token. Regular number tokens
are divided up into various types (e.g. hex, floating-point, etc.), but
pp-numbers are general enough to encompass all of them at once without
specifically knowing about any of them. As an example, the preprocessing token
0__a is a valid pp-number. Eventually, preprocessing tokens get converted into
regular tokens (as the result of preprocessing when they are passed on to the
parser and immediately prior to evaluating the controlling expression in an #if
or #elif). Some preprocessing tokens cannot be converted to regular tokens
(such as 0__a), and if that conversion is attempted, it is an error. Note that
not all preprocessing tokens are converted--only those that are passed on to the
parser (etc.). E.g.

#define EAT(x)

EAT( )

The marked part is a single pp-number preprocessing token, but because it
doesn't appear in the output of the preprocessor, no attempt should ever be make
to convert it to a regular token. The point being, that preprocessing tokens
that cannot be converted can still be used without it being an error--which is
probably *why* preprocessing tokens exist as a separate token type. What this
means for an implementation is that pp-number preprocessing tokens, unlike the
numeric tokens in the underlying language, cannot be (exclusively) stored as
numeric values, they are more like identifiers or string-literals.

Paul Mensonides

Boost list run by bdawes at, gregod at, cpdaniel at, john at