RE: [boost] Re: missing ## ... juxtaposition?

From: Paul Mensonides (pmenso57_at_[hidden])
Date: 2003-12-02 19:19:07

> [mailto:boost-bounces_at_[hidden]] On Behalf Of Dan W.

> Subsequently, I exchanged messages at the Digital Mars forum. I am no 
> expert in C++ myself, so I'm just trying to be helpful in 
> solving this 
> problem by acting as dumb messenger. I'll just copy and paste 
> a summary of 
> the DM forum dialog below.

Hi Dan.

> --------------------------------------------------------------
> ------------------------
>  > In article <bq8u65$obq$1_at_[hidden]>, Walter 
> says...  >I don't see how that derives from the spec. The 
> only token concatention  >allowed by the spec is with ##, 
> extra whitespace should be quite irrelevant.  >## is defined 
> as token concatenation, so I don't see how that is undefined.
> --------------------------------------------------------------
> ------------------------
> Even if 'concatenation' per-se is not called for, and against 
> the Standard, could it be that the "." (dot) relieves the 
> preprocessor from 
> responsibility for
> adding a space at the end of the preceding string (since the 
> dot already 
> acts as
> a kind of 'separator'..)?

No.  The preprocessor does not "insert spaces" *ever*.  At this point in
translation, the preprocessor is operating on preprocessing tokens, not
characters.  There is a big difference between a lack of whitespace and
concatenation.  The first simply has adjacent preprocessing tokens, while the
second forms a new preprocessing token.  E.g.

#define ID(x) x

#define MACRO(a, b) ID(a)b

MACRO(+,+)

results in two immediately adjacent '+' preprocessing tokens.  There is no
intervening whitespace.  Whether or not whitespace exists is irrelevant for all
purposes *except* stringizing and the creation of an <h-char-sequence>.

A preprocessor that does text stream -> text stream must insert whitespace in
order to avoid the errant retokenization that would occur when the result gets
reprocessed by some other tool (such as a C or C++ compiler).  However, that is
just a hack to make it work similarly in the presence of retokenization which
does not exist in the phases of translation.

> I just find it hilarious how the boost libraries work with so 
> many compilers, but only need dozens of ## in many files to 
> work with DM.  I wouldn't be surprised at all that they'd be 
> all wrong; --won't be the first time that everybody is wrong, 
> but this bug may be just about ready for acceptance by 
> ANSI/ISO/whatever...  ;-)

I wish that arbitrary token-pasting was well-defined.  However, the example
given doesn't even make sense (per se).  The reason is that token-pasting occurs
prior to rescanning, so a construction like this:

#define A(x) B(x) ## .h
#define B(x) x

The period (.) gets concatenated to right parenthesis before the expansion of
B(x).  Even if arbitrary token-pasting was well-defined, the argument 'x' could
contain any amount of whitespace, and cause the construction to not work
properly:

#define EMPTY()

A(file EMPTY()) // file .h

In other words, there are only certain points in which whitespace is removed or
when whitespace is condensed to only a single whitespace.  This is not one of
them.  As I said before, however, this kind of problem only occurs during
stringizing and during the creation of a header-name preprocessing token of the
form <h-char-sequence>.

Further, there is only one sure-fire way to guarantee that no whitespace exists
and that is to concatenate to a placemarker preprocessing token ala C99:

#define NO_LEADING(x) NO_LEADING_I(, x)
#define NO_LEADING_I(p, x) p ## x

#define NO_TRAILING(x) NO_TRAILING_I(, x)
#define NO_TRAILING_I(p, x) x ## p

#define NO_LEADING_AND_TRAILING(x) \
    NO_LEADING(NO_TRAILING(x)) \
    /**/

...but that is not currently well-defined in C++ as it is in C99.

> --------------------------------------------------------------
> ------------------------
>  >The separator inserted by dmc is to make the preprocessor 
> work right, it  >isn't easilly removed. I don't really 
> understand why boost seems to want to  >rely on the 
> 'juxtaposition-equals-concatenation' kludge, the ## operator 
> was  >added to Standard C specifically to move away from that 
> practice.

Juxtaposition is not concatenation, and a preprocessor that is operating at the
character level rather than the preprocessing token level at this point in
translation has to jump through hoops to mimic the behavior the actual phases of
translation.  This is not a kludge on Boost's side, this is a preprocessor
implementation kludge revolving around textual representation at a phase of
translation where it doesn't exist.

> --------------------------------------------------------------
> ------------------------
> Maybe if someone could paste the section of the Standard 
> dealing with this, 
> I'd much appreciate it.
> Yours.
> dan

There is no section of the standard that *ever* says whitespace should be
inserted.  There are only places where it says whitespace should be removed or
adjacent whitespace should be condensed.

Regards,
Paul Mensonides