Re: [boost] [Preprocessor] Adding variadic macros support

Subject: Re: [boost] [Preprocessor] Adding variadic macros support
From: Paul Mensonides (pmenso57_at_[hidden])
Date: 2010-11-27 23:07:37

On Sat, 27 Nov 2010 16:55:08 -0500, Matt Calabrese wrote:

> Also, while Microsoft has said in the past that they have no plans to
> fix their preprocessor I wouldn't consider that to be the final word, no
> matter how many times they say it. Perhaps if a library like Chaos were
> in Boost their sentiment would change -- it wouldn't be the first time
> Boost has had such an effect. I understand that they don't want to break
> code that used to work, but even that can be dealt with if they really
> wanted to. For instance, I'd imagine that it should be possible for them
> to make pragmas that affect how macro definitions would be treated by
> the preprocessor. By default everything would work in the old, broken
> way, but if a library like Chaos were to need more compliant
> preprocessing then it could "push" a flag specifying more compliant
> behavior before its macro definitions and then "pop" at the end of the
> header. Any macro definitions that were written with the compliant
> preprocessor activated would work as expected when invoked (regardless
> of what flags are set at the call-site) and any macros that were written
> without it would exhibit the old, broken behavior upon invocation,
> meaning that everyone's old code would still work right out of the box.
> This also lets people use their non-compliant code in the same
> translation unit as a library like Chaos without the user ever having to
> deal with specifying different compiler options or pragmas himself.
> Whether or not Microsoft would actually take the time to do something
> such as that is questionable, but until a library like Chaos is in Boost
> I'd imagine the chances are much less.

It most likely wouldn't be quite that simple.  (Discounting the work 
necessary to implement correct semantics in the first place.)  The symbol 
table structure would most likely have to change, for example.  
Ultimately, I think that the means of definition would be virulent WRT to 
everything except entirely self-contained stuff in a library's 
implementation.  I.e. marking macro definitions themselves as working one 
way or another depending on their context of definition would be a 
disaster.

> Out of curiosity, just how much of Chaos can VC++ handle at this point?
> Is it really entirely unusable as in almost nothing is supported? Is it
> all of the looping constructs?

It is certainly all of the looping constructs, but generally-speaking, 
almost everything.  Maybe a few concatenation macros and the like work 
properly (in the right context).

> If lambdas are syntactically easier to use I would still encourage you
> to
>> keep them in Chaos. While I am never against techniques which increase
>> compile-time speed I strongly feel that ease of use in programming is
>> far more important than waiting longer for a compilation to finish.
>>
>>
> Again, seconding this. Unless we are talking about an order of magnitude
> difference I'd imagine that the use of lambdas outweigh the cost.

In general, they are more expensive per invocation.  They are not 
asymptotically more expensive.  There are things that can be done to 
mitigate that, but I'm not doing it ATM.  The way that the lambda 
mechanism in Chaos works is that it translates the expression into an IL 
form, performs argument substitution, and then "executes" the result.  
Right now, the initial translation is being done every time, but it could 
be done once at the beginning of an algorithm.

Take a look at the result of the following:

// test.cpp

#include <chaos/preprocessor.h>

#include CHAOS_PP_PLACEHOLDERS(1)

#define e \
    CHAOS_PP_CAT_(_0, CHAOS_PP_CAT_(_1, _2)) \
    /**/

// unparsed expression...
e

// parsed (and substituted) expression...
CHAOS_PP_PARSE(
    e, A, B, C
)

// "executed" expression...
CHAOS_PP_EXECUTE(
    CHAOS_PP_PARSE(
        e, A, B, C
    )
)

// final result...
CHAOS_PP_EXPR(
    CHAOS_PP_EXECUTE(
        CHAOS_PP_PARSE(
            e, A, B, C
        )
    )
)

g++ -E -std=c++0x -I $CHAOS_ROOT -D CHAOS_PP_VARIADICS test.cpp

The parse effectively flattens all of the nesting out of the expression.  
In this implementation, however, it is also doing argument substitution.  
However, it doesn't _have_ to do that.

> Particularly when prototyping, it is a huge advantage to be able to
> quickly write concise code that works. I've been doing a relatively
> large amount of preprocessor metaprogramming lately (the libraries I've
> been working on, AutoFunction and Generic, are now about 17,000 lines
> combined, the vast majority of which are internal macros used in the
> implementation of just a handful of top-level macros) and I'd likely be
> able to go much faster with a library such as Chaos. Once all my tests
> were passing I could go back and replace lambda expressions with
> hand-written macros if compile times proved to be a problem.

I pretty much guarantee that compile times would be significantly 
higher.  Furthermore, I don't think that a library author could ever 
justify increasing compile times that signficantly for a little sugar.  I 
can see it in application code, but not in libraries--and libraries tend 
to be the place where code generation is more applicable.

> At the
> moment I find myself constantly rewriting small portions of code during
> development that ends up being fairly tedious due to the amount of
> boiler-plate that is necessary when working with Boost.Preprocessor.

Can you give me an example of the boilerplate which you are referring to?

> So, please, submit Chaos for review. I don't know how many other people
> feel about the problem of supporting non-compliant compilers, but I
> think the benefits are too numerous for a library like Chaos, especially
> given that preprocessor metaprogramming is sort of niche anyway.

I'd rather have a stronger feeling for the answer that question before 
bothering with a review process.  Here's the basic questions:

1) What happens when a library cannot be implemented on a broken 
compiler--particularly a major (and otherwise fairly decent) compiler 
like VC++?

2) A fundamental constraint of Chaos is that it is supposed to be a 
_clean_ reference implementation--i.e. no workarounds at all, not just no 
major workarounds.  The idea is to push the boundaries of what can be 
done, not "fix" vendor's compilers for them by way of hacks.  Workarounds 
of any type compromise this goal.  Once a workaround is in place, the 
interface gets used and has to be supported.  This restricts the freedom 
of the implementation, provides no incentive for compilers to be fixed, 
and ultimately ends the lifetime of the library.

3) If Boost were to internally use Chaos, the build process would have to 
include a bunch of header generation (using a Wave driver, for example, 
that emulates a compiler).