RE: [boost] Re: Some remarks on PP docs

From: Paul Mensonides (pmenso57_at_[hidden])
Date: 2004-06-11 00:50:00

> -----Original Message-----
> [mailto:boost-bounces_at_[hidden]] On Behalf Of David Abrahams
> 
> "Vesa Karvonen" <vesa_karvonen_at_[hidden]> writes:

> >>- Fundamental PP Abstractions
> >>   - Macros
> >>     - Non function
> >>     - Function macros
> >>   - Tokens
> >>   - Sequences of tokens not including un-parenthesized commas

To the macro expansion mechanism, there are two main categories of preprocessing
tokens--functional and non-functional.  The functional preprocessing tokens are
comma and the left and right parentheses.  All others are non-functional (as
input to macros).  You have to be careful whenever you pass preprocessing tokens
that are "functional" as arguments because they can have syntactic/semantic
effect.  Also note that an unmatched parenthesis can be considered pathological
input.  It is only possible to pass such a token by the use of a macro
expansion, as in MACRO(LPAREN()).  Unmatched parentheses in particular should
*never* be passed to pp-lib primitives.

> > Data structures are an important topic and their 
> representation isn't 
> > fundamentally PP *library* specific.
> > You might want to present the information on the representation of 
> > data structures (and simple macros like a projection operator for 
> > tuples and some sequences operations) here when you discuss 
> > fundamental PP abstractions.
> 
> I am not even trying to cover the whole PP library in this 
> appendix, much less the topic of PP metaprogramming in 
> general.  I am only covering the fundamental abstractions on 
> which the PP operates in this section, not the things we can 
> build with those.  The PP doesn't know anything about data structures.

Not general purpose data structures, no, but it has its own which are similar in
some respects, but not really.  You are right when you say that you can't cover
the entire pp-lib.  That is another book.  Covering preprocessor metaprogramming
in general is another (large) book beyond that.

> >>Well, if you really have a conforming C99 preprocessor, you 
> could use 
> >>Paul M's Chaos, which I think is both easier to use and *way* more 
> >>efficient than the current Boost PP.

You don't need a conforming C99 preprocessor, just a conforming C, C++, or C99
preprocessor.  With variadics enabled (i.e. C99), Chaos is certainly more
powerful than without, but they aren't strictly necessary to utilize Chaos.

> > From what I know, Chaos is essentially an extended Boost.PP 
> (meaning 
> > that Chaos is essentially a combinator library), with a single 
> > recursion backend (enabled by stricter requirements on 
> preproprecessor 
> > conformance),

The recursion backend in Chaos is more than capable of handling any real-world
example.  However, one of the things that I have on my list for version 2 is
fully generic algorithms capable of having drop-in replacements for recursion
backend, etc..  I haven't done it already despite the ease of such a thing
because I had to stop developing code and start writing documentation or I would
never stop. :)

> which makes it more orthogonal. However, Chaos still 
> > shares many problems with Boost.PP, the main issues being that code 
> > will be highly verbose and very intricate.

Chaos, in general, is not particularly verbose, only the names are verbose
because of prefixing and all-caps.  The verbosity of preprocessor code is also
significantly lessened by things that Chaos does that the pp-lib does not--such
as lazy evaluation.  OTOH, algorithm definition (but not use) is indeed
intricate--especially when doing complex things such as trampolining
user-defined macro invocations, making exponential use of the recursion backend,
supporting lambda call semantics, supporting default arguments, supporting
argument remapping, etc., all at one time (which is what Chaos does).  So yes,
the Chaos algorithms are both complex and powerful--they are feature rich.
However, simple algorithms are easy to write without the code bloat of a
Boost.PP-like implementation:

// interface

#define REPEAT(count, macro, data) \
    REPEAT_S(CHAOS_PP_STATE(), count, macro, data) \
    /**/
#define REPEAT_S(s, count, macro, data) \
    REPEAT_I( \
        CHAOS_PP_OBSTRUCT(), CHAOS_PP_NEXT(s), \
        count, macro, data \
    ) \
    /**/

// implementation

#define REPEAT_INDIRECT() REPEAT_I
#define REPEAT_I(_, s, count, macro, data) \
    CHAOS_PP_EXPR_IF _(count)( \
        CHAOS_PP_EXPR_S(s) _(REPEAT_INDIRECT _()( \
            CHAOS_PP_OBSTRUCT _(), CHAOS_PP_NEXT(s), \
            CHAOS_PP_DEC(count), macro, data \
        )) \
        macro _(s, CHAOS_PP_DEC(count), data) \
    ) \
    /**/

Is this "highly verbose"?  Compared to code that has no need to simulate
recursion, yes.  Compared to Boost.PP's version, no.  The above is also much
more powerful than pp-lib version (though much less powerful than the actual
Chaos version).  Compared to an implementation directly implemented on top of
the continuation machine... it is certainly less terse, but it is also much
clearer and significantly less design-altering.  Compared to code executed by an
interpreter such as Order's front end, yes (at the cost of a significant
performance hit for non-trivial problems).

Further, the purpose of abstractions is to avoid the need for users to implement
such things.  However, if a user wants to take that step--for the purposes of
raw speed or just interest--Chaos offers all the low-level tools necessary to
achieve it.

> The verbosity 
> and intricacy 
> > probably do not matter that much when one only performs simple 
> > repetition using the canned library macros.
> 
> Yes, I know that Chaos is not as beautiful as Order ;-)

Chaos has far more low-level components than Order--and is far more powerful in
total.  Order itself can be implemented with Chaos, but not vice-versa.
(Further, Order has to choose between providing beauty and safety regarding its
use of unprefixed names.  The safe approach has to use "8x" (or similar)
prefixed names and hide them with emacs scripts--which is not much different
than Chaos and "CHAOS_PP_".  The unsafe approach effectively assumes ownership
of a bunch of unprefixed names.)

> Paul didn't implement continuations or many of the other 
> high-level abstractions you have.

Chaos actually does have a continuation machine, similar to the one in Order.
(The original implementation was indeed Vesa's.)  Chaos also has different forms
of higher-level abstractions as well as low-level primitives.  Chaos' primary
purpose (other than as my own idealistic implementation) is to generate source
code, not provide a programming language.  Of course, the result is like a
programming language in the sense that it is utterly different from regular C or
C++ programming with different techniques and idioms.  However, the purpose is
still only to generate the real language underneath.  Chaos also has a
fundamentally different point of view.  Chaos, as opposed to Order,
intentionally encourages better understanding of the actual language being used
(which is not Chaos, but the preprocessor) rather than trying to hide it as much
as possible.  Completely hiding the nature of the interpreter is impossible and
many times the ability to work at a "lower-level" is necessary.  That isn't to
say that both points of view aren't valid, they are just different.  For
example, Order and Chaos can invoke each other--meaning that they aren't
mutually exclusive.

> On the other hand, it is 
> efficient without having to pass around recursion depths IIUC.

It is fairly efficient, but it isn't as efficient as possible--primarily because
the algorithms in Chaos are full-featured.  However, Chaos can be used to
produce the most efficient possible implementation--which sometimes means
(depending on the nature of the problem to be solved) direct use of the
continuation machine or recursion backend.

Regards,
Paul Mensonides