Re: [boost] BOOST_PP_IS_*ARY questions (Re: New Borland compiler has a conforming preprocessor?)...

From: Paul Mensonides (pmenso57_at_[hidden])
Date: 2006-01-23 00:43:43

> -----Original Message-----
> From: boost-bounces_at_[hidden] 
> [mailto:boost-bounces_at_[hidden]] On Behalf Of Tobias Schwinger

> > Yes--with regard to what I say below.  Those macros detect 
> > "parenthetic expressions"--which are non-pathological (see below) 
> > sequences of preprocessing tokens that begin with something 
> that's parenthesized.  E.g. these work also:
> > IS_UNARY((+) -) IS_UNARY((+)(not, unary))
> 
> Non-pathological means "for every opening parentheses there's 
> a closing one" (I guess it applies at least to the outer 
> nesting level)? Correct?

Well, for all nesting levels really.

Many arguments to library macros are typed in the sense that they must be within
some domain of input.  E.g. the number of repetitions, a macro compatible with a
particular signature, etc..  For those arguments, the input must correspond to
the domain.  Other arguments are untyped with a few domain restrictions (not yet
getting into pathological input).  IS_UNARY's argument is one of these--you can
pass anything to it except a parenthetic expression of a different arity.  E.g.
IS_UNARY((1, 2)) is illegal input and will cause an error.  Other arguments are
untyped with no restrictions--except pathological input.  Pathological input
consists of passing half-open parentheses (only possible through indirect macros
like LPAREN and RPAREN) and commas in certain situations.  These tokens are
functional operators to the preprocessor, and they can foul up all of the
internals of a macro definition.  E.g.

#define A(m, args) m(args)

A(MACRO, RPAREN())
-> m())
-> probably an error

A(MACRO, LPAREN())
-> m(()
-> probably an error

A(MACRO, COMMA())
-> m(,)
-> probably an error

That is what I mean by pathological input, and you can pass it at any level of
scanning.  Lastly, some arguments are untyped an have no restrictions at
all--including pathological input.  These are a precious few, however--e.g.

#define ID(x) x

ID(LPAREN())

In general, because of the way that the preprocessor works, you simply cannot
design macros to deal with pathological input.

> > What compilers/preprocessors do you usually use?
> > 
> 
> I'm interested in writing compiler independent portable C++ 
> code that works in practice ;-)...

Then I'd avoid directly using those macros.  They are perfectly legal, but even
though they should work portably and consistently, they don't.

> > but only sometimes (IOW, they are unstable).  In the latter 
> case, it 
> > isn't just a simple case of "if you pass it this, it doesn't work".
> 
> Do these problems (besides the ones with BCC) apply to 
> pathological token sequences only?

No, it is more like how a particular sequence of tokens is constructed.  I'll
try to explain what I'm referring to better...  Say you have some token sequence
'()' and you pass it to IS_NULLARY, which ultimately results in '1' (as it
should).  Say then you have some other macro M that ultimately results in '()'.
You test this macro, and it works.  The you try IS_NULLARY(M()) and that works.
Then, you have yet another macro N that also ultimately results in '()' (and it
does what it should), but when you try to pass it to IS_NULLARY(N()) it suddenly
doesn't work--and not because of particular names being disabled (i.e. it works
fine on a conformant preprocessor).  It is suddenly related to *how* a
particular sequence of tokens is constructed.  Some preprocessors seem to do
things in strange orders or delay doing certain things until they *think* the
results are necessary as some sort of optimization.  Because the assumptions
about how these changes will affect accurate preprocessing results are wrong, it
causes all kinds of problems.  That is *precisely* the problem with these macros
on several popular preprocessors.

> > VC configuration, for example).  The result for client use 
> is highly 
> > unpredictable because client code doesn't contain the 
> scaffolding (the pp-lib bends over backward to minimize the 
> this in user code).
> 
> The only two places where is_*ary seem to be used are:
> 
>    list/adt.hpp
>    facilities/apply.hpp
> 
> I can't see no scaffolding in the VC-case there. Am I looking 
> in the wrong spot?

I'm referring to the sorts of manipulation that these macros do more than the
macros themselves.  Part of why they aren't used more often is because they can
be unstable and part of it is because the "other way to do it that requires more
macros" is already in place because of preprocessors where they don't work at
all.

Take a look at 'cat.hpp'.  It is a good example of the kind of scaffolding
required for VC++ and Metrowerks (before they rewrote their preprocessor).  That
scaffolding is not really about making CAT work properly, it is about making 1)
input to CAT work properly and 2) input that uses CAT work properly to other
macros--such as IS_UNARY.  The library does this kind of thing all over the
place, and it has a kind of stabilizing effect on the net result (though it
isn't a complete solution).  Client code, however, doesn't do this (and it
shouldn't have to--neither should the library, really).  How stable the library
is in general on horribly broken preprocessors (like VC++) is a testament to how
much effort the library puts into forcing a degree of stabilization.

> > On
> > reasonably good preprocessors, this isn't a problem at all, 
> and using 
> > them is perfectly safe and predictable.
> > 
> 
> Would you mind listing the "reasonably good" ones (that is, 
> if there are more on the list than GNU and Wave)?

    Wave
    Unicals
    GCC
    EDG's new preprocessor
    Digital Mars' new preprocessor
    Metrowerks' new preprocessor
    (probably Borland's new preprocessor)

    ...and there are others that I've seen
    embedded in other source processing tools,
    and I might be forgetting a few off the top
    of my head.

This list doesn't mean that these preprocessors are perfect, but they are
generally pretty good.  EDG's new preprocessor is not yet available in a version
of Comeau (I have a prerelease beta.) and I'm not sure if it is availabe in a
new release of Intel's compiler.  I have a prerelease beta that corrects several
issues with Digital Mars' new preprocessor, but I don't know if those
corrections have made it into an official release yet.

> > Regarding your DO_UNARY-esque macros...  They would be more 
> general if 
> > they were something like:
> >  [...]
> 
> Sorry for picking over-generalized examples (in fact, there 
> was no intention to provide a general solution). They were 
> intended to illustrate why these IS_*ARY macros would be a 
> great for (very dedicated and non-generic ;-) ) user code.

Yes.

> > BTW, with variadics, you can do all kinds of fun things related to 
> > optional/default arguments or overloading based on number of 
> > arguments.  Some simple examples...
> > 
> > [... code]
> 
> Amazing! Of course I wouldn't ask questions about using 
> IS_*ARY for optional arguments if all preprocessors would 
> support these extensions. Until then I will most hopefully 
> find the time for an in-depth study of Chaos!

Be warned--the implementation of much of Chaos is extremely complex.  It isn't
difficult to use (in most cases) however.  The parts where it can be difficult
to use are the parts that are exposed as interfaces that allow users to
construct components similar to those provided by the library itself.  I.e. the
philosophy is something like, "If component XYZ is generalizable enough to be
used by the library itself in multiple places, then it should be exposed as a
proper interface so that client code can use if desired."  The detection macros
like IS_UNARY are perfect examples of that--they are all proper, documented
library interfaces (though there are other things that are far more complex).
In other words, Chaos doesn't just provide you with tools that have complex
implementations but solve simple or common tasks, it also provides you with
tools to make the complex implementations themselves (Chaos reuses itself a lot,
which is something that the pp-lib doesn't do.).

> BTW. are both variadic macros and overloading part of C99?

Variadics macros and placemarkers are part of C99 (and part of the next C++
standard when it is released).  Overloading can be simulated with those
things--which is what Chaos does.  Even without them, however, Chaos is still
significantly more powerful than the pp-lib, and with them, it is significantly
more powerful than without.  The downside of Chaos is that it cannot be
implemented without a very good preprocessor and therefore isn't as portable as
the limited subset of possible functionality provided by the Boost pp-lib.

Things are improving, slowly but surely--just look at the number of "new
preprocessor" instances in the list of preprocessors above.  Plus, both Wave and
Unicals are relatively new.  For Boost, there are three significant hurdles
left:  Microsoft's, Sun's, and IBM's preprocessors.  VC++ is *by far* the
biggest hurdle and the most broken "modern" preprocessor that I've seen
(Metrowerks used to hold that honor).  Though VC++ is probably the most popular
compiler, the significance of Boost's support for Sun and IBM can't be ignored.
If at some point, Chaos or something like it (say upgraded Boost.PP) was to
become part of Boost, no Boost libraries could really use it until VC++ is fixed
because supporting VC++ is too important.

Regards,
Paul Mensonides