Re: [boost] [Preprocessor] Adding variadic macros support

Subject: Re: [boost] [Preprocessor] Adding variadic macros support
From: Edward Diener (eldiener_at_[hidden])
Date: 2010-11-29 09:55:40

On 11/29/2010 7:58 AM, Paul Mensonides wrote:
> On Sun, 28 Nov 2010 08:08:15 -0500, Edward Diener wrote:
>
>>> I didn't know that there was a tar.gz file.  AFAIR, I've never put one
>>> up.  The project has existed since before Sourceforge had Subversion
>>> services.  At that time, Boost was hosted there in a CVS repository.
>>
>> If you could get the Chaos files in a subversion repository I think it
>> would be helpful. I have nothing in particular against CVS other than
>> the fact that like most programmers I find subversion much more flexible
>> to use.
>
> I was planning on doing version 2 in the Subversion repository there.  I
> also prefer SVN over CVS, but I also don't see any immediate gain by a
> switch over.  This is particularly true as Chaos is a one developer
> project.

The gain is only that people interested in Chaos will find it easier to 
obtain the latest code if, like me, they largely forego the use of CVS. 
I do admit that there are still plenty of other projects on the Net that 
still use CVS, so maybe I should just give in and install a CVS front end.

>
>> In a way the lambda use above is syntactically easier because one does
>> not have to break up the usage into two pieces. Often syntactically
>> easier does not mean shorter but more understandable in the way
>> something is constructed.
>
> In some ways, yes.
>
>> It is up to you whether or not you find lambda syntax within Chaos
>> easier to use and/or flexible enough to justify their inclusion in
>> Chaos. My general point is that unless compiler time increases to a very
>> large extent when creating C++ constructs, I find that programmers
>> complaining about excess compiler time when using C++ ( or any other
>> language ) are off base. In today's programming world valuable
>> programming work can almost always be done while a programmer waits for
>> a compile/build to finish.
>
> Well, here are some concrete timings.  The following are two source files
> that do the same thing.  They generate a binary sequence of pairs of the
> following form (1, n)(2, n - 1) ... (n - 1, 2)(n, 1) where n equals 100.
> (This is not interesting, it merely gives opportunity use multiple
> lambdas.)  The first source file does everything with lambdas.  The
> second source file does everything without lambdas.
>
> // 1.cpp
>
> #include<chaos/preprocessor.h>
>
> ??=include CHAOS_PP_PLACEHOLDERS(1)
>
> #define ZIP(a, b) \
>      CHAOS_PP_VARIADIC_ELEM( \
>          0, \
>          CHAOS_PP_EXPR(CHAOS_PP_SEQ_FOLD_LEFT( \
>              _2 (_1, CHAOS_PP_SEQ_HEAD_(_3)) CHAOS_PP_COMMA_() \
>                  CHAOS_PP_SEQ_TAIL_(_3), \
>              a,, b \
>          )) \
>      ) \
>      /**/
>
> #define NATURALS(n) \
>      CHAOS_PP_EXPR(CHAOS_PP_REPEAT( \
>          n, \
>          (CHAOS_PP_INC_(_1)) \
>      )) \
>      /**/
>
> #define GAUSS(n) \
>      GAUSS_I(NATURALS(n)) \
>      /**/
> #define GAUSS_I(seq) \
>      ZIP( \
>          seq, \
>          CHAOS_PP_SEQ_REVERSE(seq) \
>      ) \
>      /**/
>
> GAUSS(100)
>
> ??=include CHAOS_PP_PLACEHOLDERS(0)
>
> // 2.cpp
>
> #include<chaos/preprocessor.h>
>
> #define ZIP(a, b) \
>      CHAOS_PP_VARIADIC_ELEM( \
>          0, \
>          CHAOS_PP_EXPR(CHAOS_PP_SEQ_FOLD_LEFT( \
>              ZIP_O, a,, b \
>          )) \
>      ) \
>      /**/
> #define ZIP_O(s, e, r, b) \
>      r (e, CHAOS_PP_SEQ_HEAD(b)), CHAOS_PP_SEQ_TAIL(b) \
>      /**/
>
> #define NATURALS(n) \
>      CHAOS_PP_EXPR(CHAOS_PP_REPEAT( \
>          n, \
>          NATURALS_M \
>      )) \
>      /**/
> #define NATURALS_M(s, n) (CHAOS_PP_INC(n))
>
> #define GAUSS(n) \
>      GAUSS_I(NATURALS(n)) \
>      /**/
> #define GAUSS_I(seq) \
>      ZIP( \
>          seq, \
>          CHAOS_PP_SEQ_REVERSE(seq) \
>      ) \
>      /**/
>
> GAUSS(100)
>
> The first file (w/lambda) takes about 49.4 seconds to preprocess with g++
> on my Linux VM.  The second file (w/o lambda) takes about 1.9 seconds to
> preprocess.

The 49.4 seconds really is slow. Despite my protestations about 
programmers complaining about compile times, if it took nearly a minute 
to compile a single source file I would find this very unusual. I am not 
sure what you mean by a Linux VM however. Are you running under VMWare, 
or something similar ? Maybe that explains the slow time as opposed to 
running natively. I will try it myself and see the difference in timing, 
but I admit I am a Windows person largely, although I have a number of 
Linux distros into which I occasionally boot, so I will try it under 
Windows with mingw/gcc.

>  That is about 26 times faster w/o lambda.  In this
> particular case, the two operations that use lambdas are both O(n), so
> use your imagination for how bad it would get if an algorithm was non-
> linear.  To me, that kind of performance drain is simply unacceptable for
> use in library code unless that code is self-contained (i.e. the library
> isn't providing macros that do this as a user-interface to clients) and
> the n is relatively small.
>
> That isn't an argument for total exclusion of the mechanism (or a similar
> mechanism), as non-library code could still use it if desired.  However,
> the main context for code generation is in libraries or in application
> code using interfaces provided by libraries (i.e. macros published as
> interfaces by libraries).  I'm simply not sure that 1) the implementation
> cost is worth the benefit, and 2) whether the tacit endorsement, by way
> of the existence of the mechanism, is a good idea.

I would guess that if you found maintaining the lambda interface to 
Chaos was too much work, particularly as you decide to add any new 
functionality to Chaos, you could drop it. Otherwise just documenting 
that it is much slower during compilation IMO would be adequate.

>
> In reference to prototyping, there is a cost there as well.  In the
> course of development, one of the things that I often do is strategically
> stop certain macros from expanding so that I can see what the partial
> results are.  For example, in the following
>
> #define SUM(seq) \
>      CHAOS_PP_EXPR(CHAOS_PP_SEQ_FOLD_LEFT( \
>          SUM_O, seq, 0 \
>      )) \
>      /**/
> #define SUM_O(x, y) CHAOS_PP_ADD(x, y
>
> SUM( (1)(2)(3) )
>
> ...I've made two errors.  The first is that I'm missing the 's' parameter
> (which is the recursion state of the library) in the SUM_O folding
> operation.  In this case, the problem is somewhat easy to find, but I can
> see what's going on just by putting a period after the SUM_O in the
> definition of SUM.
>
> #define SUM(seq) \
>      CHAOS_PP_EXPR(CHAOS_PP_SEQ_FOLD_LEFT( \
>          SUM_O., seq, 0 \
>      )) \
>      /**/
> #define SUM_O(x, y) CHAOS_PP_ADD(x, y
>
> SUM( (1)(2)(3) )
>
> This yields
>
> =>  SUM_O.(3, 3, SUM_O.(3, 2, SUM_O.(3, 1, 0)))

I have never seen this technique before for debugging macros. Is this 
something built-in to Chaos or something the compiler reacts to ?

>
> Given that I'm familiar with Chaos' recursion mechanisms, I immediately
> spot my error and fix it.
>
> #define SUM(seq) \
>      CHAOS_PP_EXPR(CHAOS_PP_SEQ_FOLD_LEFT( \
>          SUM_O, seq, 0 \
>      )) \
>      /**/
> #define SUM_O(s, x, y) CHAOS_PP_ADD(x, y
>
> SUM( (1)(2)(3) )
>
> This time, however, I run afoul of my second error, the missing closing
> parenthesis in the invocation of ADD.  Here, GCC's error messages are
> even more unhelpful.  I get
>
> =>  unterminated argument list invoking macro "CHAOS_IP_EXPR_3"
>
> However, I know that the fold is generated correctly because of the
> debugging process for bug #1.  Therefore, I strategically stop the
> expansion of ADD.
>
> #define SUM(seq) \
>      CHAOS_PP_EXPR(CHAOS_PP_SEQ_FOLD_LEFT( \
>          SUM_O, seq, 0 \
>      )) \
>      /**/
> #define SUM_O(s, x, y) CHAOS_PP_ADD.(x, y
>
> SUM( (1)(2)(3) )
>
> However, I get the exact same error.  Therefore, I know that the error is
> not occurring because ADD is getting nonsensical input.
>
> Eventually, I figure out my obvious problem and fix it.
>
> The issue here is not really this particular example so much as it is
> debugging techniques with preprocessor stuff.  This example is trivial
> and obvious.  Other times, the result is a mountain of garbage for output
> that makes template error messages look readable.  (I'm sure anybody that
> has done any significant preprocessor generation has seen this.)

Whenever I have gotten error messages from incorrect macros, using Boost 
PP, I just break down the macro from the inside as much as possible, 
into test pieces, and turn on the compiler switch that gives me 
proprocessed output.

>  Some of
> the algorithms in Chaos make exponential use of the recursion backend in
> such a way that if machine memory was unlimited, it would literally take
> several millenia to finish printing the output to the screen.  It is
> techniques like those above that can be used to isolate bug locations,
> condense and structure output, or make the output finite (in a practical
> sense) thus producing something that has a hope of being deciphered.
>
> When lambda is involved, you simply cannot do things like that that
> easily.  Because of that, whenever I'm illustrating the lambda mechanism,
> I virtually always prototype whatever I'm doing w/o lambda and then port
> it.

Understood. Thanks for the explanation.