Re: [boost] [Preprocessor] Adding variadic macros support

Subject: Re: [boost] [Preprocessor] Adding variadic macros support
From: Paul Mensonides (pmenso57_at_[hidden])
Date: 2010-11-29 07:58:42

On Sun, 28 Nov 2010 08:08:15 -0500, Edward Diener wrote:

>> I didn't know that there was a tar.gz file.  AFAIR, I've never put one
>> up.  The project has existed since before Sourceforge had Subversion
>> services.  At that time, Boost was hosted there in a CVS repository.
> 
> If you could get the Chaos files in a subversion repository I think it
> would be helpful. I have nothing in particular against CVS other than
> the fact that like most programmers I find subversion much more flexible
> to use.

I was planning on doing version 2 in the Subversion repository there.  I 
also prefer SVN over CVS, but I also don't see any immediate gain by a 
switch over.  This is particularly true as Chaos is a one developer 
project.

> In a way the lambda use above is syntactically easier because one does
> not have to break up the usage into two pieces. Often syntactically
> easier does not mean shorter but more understandable in the way
> something is constructed.

In some ways, yes.
 
> It is up to you whether or not you find lambda syntax within Chaos
> easier to use and/or flexible enough to justify their inclusion in
> Chaos. My general point is that unless compiler time increases to a very
> large extent when creating C++ constructs, I find that programmers
> complaining about excess compiler time when using C++ ( or any other
> language ) are off base. In today's programming world valuable
> programming work can almost always be done while a programmer waits for
> a compile/build to finish.

Well, here are some concrete timings.  The following are two source files 
that do the same thing.  They generate a binary sequence of pairs of the 
following form (1, n)(2, n - 1) ... (n - 1, 2)(n, 1) where n equals 100.  
(This is not interesting, it merely gives opportunity use multiple 
lambdas.)  The first source file does everything with lambdas.  The 
second source file does everything without lambdas.

// 1.cpp

#include <chaos/preprocessor.h>

??=include CHAOS_PP_PLACEHOLDERS(1)

#define ZIP(a, b) \
    CHAOS_PP_VARIADIC_ELEM( \
        0, \
        CHAOS_PP_EXPR(CHAOS_PP_SEQ_FOLD_LEFT( \
            _2 (_1, CHAOS_PP_SEQ_HEAD_(_3)) CHAOS_PP_COMMA_() \
                CHAOS_PP_SEQ_TAIL_(_3), \
            a,, b \
        )) \
    ) \
    /**/

#define NATURALS(n) \
    CHAOS_PP_EXPR(CHAOS_PP_REPEAT( \
        n, \
        (CHAOS_PP_INC_(_1)) \
    )) \
    /**/

#define GAUSS(n) \
    GAUSS_I(NATURALS(n)) \
    /**/
#define GAUSS_I(seq) \
    ZIP( \
        seq, \
        CHAOS_PP_SEQ_REVERSE(seq) \
    ) \
    /**/

GAUSS(100)

??=include CHAOS_PP_PLACEHOLDERS(0)

// 2.cpp

#include <chaos/preprocessor.h>

#define ZIP(a, b) \
    CHAOS_PP_VARIADIC_ELEM( \
        0, \
        CHAOS_PP_EXPR(CHAOS_PP_SEQ_FOLD_LEFT( \
            ZIP_O, a,, b \
        )) \
    ) \
    /**/
#define ZIP_O(s, e, r, b) \
    r (e, CHAOS_PP_SEQ_HEAD(b)), CHAOS_PP_SEQ_TAIL(b) \
    /**/

#define NATURALS(n) \
    CHAOS_PP_EXPR(CHAOS_PP_REPEAT( \
        n, \
        NATURALS_M \
    )) \
    /**/
#define NATURALS_M(s, n) (CHAOS_PP_INC(n))

#define GAUSS(n) \
    GAUSS_I(NATURALS(n)) \
    /**/
#define GAUSS_I(seq) \
    ZIP( \
        seq, \
        CHAOS_PP_SEQ_REVERSE(seq) \
    ) \
    /**/

GAUSS(100)

The first file (w/lambda) takes about 49.4 seconds to preprocess with g++ 
on my Linux VM.  The second file (w/o lambda) takes about 1.9 seconds to 
preprocess.  That is about 26 times faster w/o lambda.  In this 
particular case, the two operations that use lambdas are both O(n), so 
use your imagination for how bad it would get if an algorithm was non-
linear.  To me, that kind of performance drain is simply unacceptable for 
use in library code unless that code is self-contained (i.e. the library 
isn't providing macros that do this as a user-interface to clients) and 
the n is relatively small.

That isn't an argument for total exclusion of the mechanism (or a similar 
mechanism), as non-library code could still use it if desired.  However, 
the main context for code generation is in libraries or in application 
code using interfaces provided by libraries (i.e. macros published as 
interfaces by libraries).  I'm simply not sure that 1) the implementation 
cost is worth the benefit, and 2) whether the tacit endorsement, by way 
of the existence of the mechanism, is a good idea.

In reference to prototyping, there is a cost there as well.  In the 
course of development, one of the things that I often do is strategically 
stop certain macros from expanding so that I can see what the partial 
results are.  For example, in the following

#define SUM(seq) \
    CHAOS_PP_EXPR(CHAOS_PP_SEQ_FOLD_LEFT( \
        SUM_O, seq, 0 \
    )) \
    /**/
#define SUM_O(x, y) CHAOS_PP_ADD(x, y

SUM( (1)(2)(3) )

...I've made two errors.  The first is that I'm missing the 's' parameter 
(which is the recursion state of the library) in the SUM_O folding 
operation.  In this case, the problem is somewhat easy to find, but I can 
see what's going on just by putting a period after the SUM_O in the 
definition of SUM.

#define SUM(seq) \
    CHAOS_PP_EXPR(CHAOS_PP_SEQ_FOLD_LEFT( \
        SUM_O., seq, 0 \
    )) \
    /**/
#define SUM_O(x, y) CHAOS_PP_ADD(x, y

SUM( (1)(2)(3) )

This yields

=> SUM_O.(3, 3, SUM_O.(3, 2, SUM_O.(3, 1, 0)))

Given that I'm familiar with Chaos' recursion mechanisms, I immediately 
spot my error and fix it.

#define SUM(seq) \
    CHAOS_PP_EXPR(CHAOS_PP_SEQ_FOLD_LEFT( \
        SUM_O, seq, 0 \
    )) \
    /**/
#define SUM_O(s, x, y) CHAOS_PP_ADD(x, y

SUM( (1)(2)(3) )

This time, however, I run afoul of my second error, the missing closing 
parenthesis in the invocation of ADD.  Here, GCC's error messages are 
even more unhelpful.  I get

=> unterminated argument list invoking macro "CHAOS_IP_EXPR_3"

However, I know that the fold is generated correctly because of the 
debugging process for bug #1.  Therefore, I strategically stop the 
expansion of ADD.

#define SUM(seq) \
    CHAOS_PP_EXPR(CHAOS_PP_SEQ_FOLD_LEFT( \
        SUM_O, seq, 0 \
    )) \
    /**/
#define SUM_O(s, x, y) CHAOS_PP_ADD.(x, y

SUM( (1)(2)(3) )

However, I get the exact same error.  Therefore, I know that the error is 
not occurring because ADD is getting nonsensical input.

Eventually, I figure out my obvious problem and fix it.

The issue here is not really this particular example so much as it is 
debugging techniques with preprocessor stuff.  This example is trivial 
and obvious.  Other times, the result is a mountain of garbage for output 
that makes template error messages look readable.  (I'm sure anybody that 
has done any significant preprocessor generation has seen this.)  Some of 
the algorithms in Chaos make exponential use of the recursion backend in 
such a way that if machine memory was unlimited, it would literally take 
several millenia to finish printing the output to the screen.  It is 
techniques like those above that can be used to isolate bug locations, 
condense and structure output, or make the output finite (in a practical 
sense) thus producing something that has a hope of being deciphered.

When lambda is involved, you simply cannot do things like that that 
easily.  Because of that, whenever I'm illustrating the lambda mechanism, 
I virtually always prototype whatever I'm doing w/o lambda and then port 
it.

Regards,
Paul Mensonides