Re: [boost] [preprocessor] Warning: Incoming

Subject: Re: [boost] [preprocessor] Warning: Incoming
From: Paul Mensonides (pmenso57_at_[hidden])
Date: 2011-06-26 20:11:49

On Sun, 26 Jun 2011 11:31:55 -0400, Edward Diener wrote:

> On 6/26/2011 10:57 AM, Lorenzo Caminiti wrote:

>> If the pp-lib needs to be changed, why not change it radically to add
>> ground-up variadic support? (This is not a rhetoric question. I am
>> sincerely asking as there might be very good reasons not to add
>> ground-up variadic support to Boost.Preprocessor.)
> 
> I am not sure what Paul meant by "ground-up" variadic support, nor how
> you take it, but the variadic support in pp-lib is very much the same as
> in my VMD library with a few added enhancements based on Paul's
> knowledge of preprocessor programming. The overall intent in pp-lib was
> the same as VMD: to integrate the use of variadic macros with pp-lib's
> richer data types ( tuple, array, list, seq ) and to enhance the use of
> pp-lib tuples with variadic macro support.

This is a good summary of the changes.

What I meant by "ground-up" would be effectively re-implementing 
Boost.Preprocessor from scratch with variadic macros and placemarkers in 
mind.  For example, with decent preprocessors (something we don't have as 
a rule), this would drastically reduce the number of use cases for 
BOOST_PP_TUPLE_ELEM.  The use case I'm specifically thinking of at the 
moment is the encoding and decoding of auxiliary data or state data 
passed through algorithms provided by the library.  E.g.

#define ADD(x, y) WHILE(PRED, OP, (x, y))

#define PRED(d, state) \
    TUPLE_ELEM(2, 1, state) \
    /**/
#define OP(d, state) \
    (INC(TUPLE_ELEM(2, 0, state)), DEC(TUPLE_ELEM(2, 1, state))) \
    /**/

This one type of scenario decreases the efficiency and drastically 
decreases the readability of the code because the TUPLE_ELEM (or similar) 
clutter required in the implementation of both PRED and OP is either 
pervasive or results in more macro definitions.

Ground-up support would imply that WHILE (and every other higher-order 
macro in the library) allow variadic state such that the example becomes:

#define ADD(x, y) WHILE(PRED, OP, x, y)

#define PRED(d, x, y) y
#define OP(d, x, y) INC(x), DEC(y)

The amount of clutter, even in this trivial example, is significantly 
reduced resulting in smaller, more efficient, and more readable code.

However, even doing this requires breaking interfaces because existing 
arguments are in the wrong orders in many places.  For example, 
SEQ_FOLD_LEFT's interface is SEQ_FOLD_LEFT(op, state, seq) where op's 
interface must be op(s, state, elem).  This would need to be SEQ_FOLD_LEFT
(op, (a1, a2, ...)(b1, b2, ...), s1, s2, ...) where op's interface must 
"compatible" with op(s, e1, e2, ..., s1, s2, ...).  In this case, that's 
variadic accumulation state (i.e. fold state) together with potentially n-
ary sequence data (in this case, not variadic sequence data such as (1)
(1, 2)(1, 2, 3), but sequence data that is unary (1)(1)(1), binary (1, 2)
(1, 2)(1, 2), or whatever so long as each element has the same "arity").

All of that aside, there is no way to generalize recursion on broken 
compilers--particularly popular ones such as VC++.  The lack of this 
leads to an implementation explosion.  Every reentrant algorithm in 
Boost.Preprocessor is implemented with a huge set of macros.  Often 
times, they are implemented with multiple huge sets of macros which are 
different in that they contain differing workarounds for various 
compilers.  Furthermore, many other higher-order algorithms are not 
reentrant because they are not implemented that way (and are instead 
implemented in terms of the ones which are).  This inherently leads to 
implementation bubbling which is particularly costly with the 
preprocessor since there is no direct recursion and because the resulting 
error messages can be as bad as template-related error messages.

With decent preprocessors, on the other hand, recursion is both 
generalizable and extensible.  For example, all higher-order algorithms 
in Chaos are reentrant and none of them are implemented via huge sets of 
algorithm-specific macros.  At the end of the day, the lack of 
generalizable recursion is even more fundamental and cripples 
Boost.Preprocessor even more than lack of ground-up support for variadic 
macros.  I consider the entire methodology of Boost.Preprocessor to be 
garbage due to widespread, aspectual workarounds.  Because of the above, 
I have no incentive to put major effort into minor improvements 
particularly when some vendor is too lazy to fix their preprocessor 
(which is a minor implementation in comparison to the underlying compiler 
and optimizer) or has too much of a "the standard is whatever we say it 
is" mentality.

Regards,
Paul Mensonides