Re: [boost] [preprocessor] Patch for stringize.hpp

From: Paul Mensonides (pmenso57_at_[hidden])
Date: 2006-02-20 15:01:37

> -----Original Message-----
> From: boost-bounces_at_[hidden] 
> [mailto:boost-bounces_at_[hidden]] On Behalf Of Steve Cornett

> Perhaps I was not clear, plus the ## operator is a trickster. 
>  Let's try
> this: you can prove to yourself that the ## operators are 
> having no effect (and are therefore unnecessary) with this 
> sample program below.  Compile and run it both with and 
> without the patch.  You will see no change in the behavior.
> 
> #include <stdio.h>
> #include "stringize.hpp"
> int main()
> {
>         puts(BOOST_PP_STRINGIZE(x));
>         return 0;
> }

Perhaps *I* was not clear.  This doesn't prove anything.  You don't understand
what the use of ## is a workaround in that code.  It isn't a workaround for any
kind of stringizing issue.  It is a workaround for Microsoft's utterly broken
macro expansion routine that does expansion at (seemingly) arbitrary times
(AFAICT, it is supposed to be some sort of optimization).  The purpose of the
use of ## is to _induce_ MS's preprocessor to finish expanding the argument.  In
a simple example like the above, you won't see the difference.  If the argument
to STRINGIZE is a complex macro expansion, you might (and you might not).

> So beyond being unnecessary, let me again attempt to convince 
> you that the ## operators are also wrong and therefore 
> undesirable.

As I said before, I know that it results in undefined behavior.  It is a
desireable, nonetheless, as a workaround for other things.  If I didn't need the
workaround, it wouldn't be there.

> First let me explain my motivation.  Our tool 
> issues a warning when our end user compiles this file with 
> our code coverage tool together with Microsoft C++.  The 
> reason for the warning is a long story, but we need it.

And the reason for the workaround is a long story--but we need it.

> Other than the warning, our behavior with the ## operator is 
> exactly the same as Microsoft C++. 
> Sometimes end users turn on the option that says all warnings 
> are errors, and then their build fails.  So we don't want to 
> bother the end user with something they have no control over.

It cannot be helped.  Those token-pasting operators must be their in the
configuration for Microsoft and the configuration for Metrowerks prior to
version 9.

> Anyway, let's look closely at this line:
> 
> #    define BOOST_PP_STRINGIZE_A(arg) BOOST_PP_STRINGIZE_B ## (arg)
> 
> In BOOST_PP_STRINGIZE_A there is an attempt to join 
> "BOOST_PP_STRINGIZE_B" 
> with "(" that would result in a single token 
> "BOOST_PP_STRINGIZE_B(".  That does not make sense, this is 
> clearly two separate tokens, an identifier and a '('.  The ## 
> is not really accomplishing anything here.  The only effect 
> of ## here is to make this macro behavior undefined by the C 
> and C++ standards.

No.  The effect is that it induces the MS preprocessor to (usually) finish the
expansion of 'arg' (which it should have done long before this point).  You
don't need to point out how it is undefined behavior--I already know--really.

> The problem is just a little harder to see in BOOST_PP_STRINGIZE_B:
> 
> #    define BOOST_PP_STRINGIZE_B(arg) BOOST_PP_STRINGIZE_I ## arg
> 
> In this case, the argument "arg" is always passed in from 
> BOOST_PP_STRINGIZE and it always begins with "(".  But after 
> that, it is the same story as before, you cannot join an 
> identifier with a '('.

Again.  I know.

> So I hope you see more clearly what is happening now.

I saw clearly (and so did Dave) exactly what you were talking about in your
first post.  The problem with the VC++ preprocessor is far deeper than the bug
you referenced via the url that you posted.  All kinds of things are broken--the
most significant of which is that the order expansion is all screwed up.  Let me
give you a concrete example:

#define IM p, q

#define A(im) B(im)
#define B(x, y) x + y

A(IM)

This expansion should result in p + q.  The _single_ argument to A should be
expanded on entry to A, the 'im' in the replacement list of A should be
substituted with the result of that expansion ( p, q ), _then_ B should be
invoked such as B(p, q).  Instead, MS's preprocessor doesn't expand 'im' where
it should, gives a warning about "not enough actual parameters for macro 'B'",
and results in:

p, q +

Now, if the workaround is present:

#define IM p, q

#define A(im) B((im))
#define B(par) C ## par
#define C(x, y) x + y

A(IM) // p + q

The concatenation, while undefined behavior, induces the expansion of 'im'
before it is passed to C, and yields the correct results.

Note that there is no other way (AFAIK) to induce the expansion.  Extra macros
don't work:

#define IM p, q

#define A(im) B(im)
#define B(im) C(im)
#define C(im) D(im)
#define D(im) E(im)
#define E(x, y) x + y

A(IM)

No matter how many macros it goes through, the preprocessor will not expand the
argument until it _thinks_ that it needs to.  The problem is that where it
_thinks_ that it needs to does not include all of the situations where actually
_does_ need to (for this and a variety of other circumstances).  The use of the
token-pasting operator makes it _think_ that it needs to.

This is only a trivial example that shows only one of the many ways that the
flawed algorithm can cause problems.  If I don't do it, the library becomes
horribly unstable on VC++ and Metrowerks (< v9)--to the point of being nearly
unusable for anything even close to complex.

> By the way, when I compile the whole boost library with our 
> tool, I  see the warning about ## producing invalid results 
> only here in stringize.hpp.

It is used in so many places it is ridiculous (e.g. 'cat.hpp',
'control/iif.hpp', etc., etc.).  It is not even *close* to the only place that
it is used.

In any case, I'm sorry, but those workarounds cannot be removed.  I don't know
what the means for your tool--though I understand the situation that you're
dealing with.  Perhaps you'll have to cause the tool to recognize Boost headers
and disable the warnings.  But, until MS fixes their preprocessor, I absolutely
*must* have those inducers in place.

Regards,
Paul Mensonides