Re: Understanding MPL

From: Paul Mensonides (yg-boost-users_at_[hidden])
Date: 2003-04-03 19:04:17

Edward Diener wrote:

> The general user does know how to use preprocessing macros and does
> understand how the macros may get expanded to code which can be used
> to accomplish tasks.

This is flat out not true.  The general user understands that macros are
relatively simple textual expansions.  The general user does not understand the
details of macro expansion, and these details are extremely important for any
effective use of the preprocessor metaprogramming.  Preprocessor metaprogramming
and, to a large extent, template metaprogramming is all about
manipulating/exploiting the "details" of their respective parts of the language.
I absolutely guarantee that this level of understanding does not generally
exist--for template metaprogramming and especially for preprocessor
metaprogramming.

> The general user does not understand the areas of
> functionality in the Boost pp-lib, what they are and how they may be
> used, because there is no good organizational explanation about the
> various areas in which the Boost pp-lib can be effectively used to
> accomplish either template programming goals or just useful code
> generation goals.

Both and more.  As I said in the post I just sent a few minutes ago, I will
address this this weekend.  I'll try to write up a decent overview of the
functionality available and send it to you so you can tell me what you think.
However, there is no getting around the complexity involved because Cpp works
against encapsulation.

> What is needed, even given the difficulty of trying
> to understand how the macros actually work with various possibly
> broken preprocessors, is an explanation of the pp-lib from a user's
> point of view, ie. what is there and what can it accomplish in terms
> of helping to simplify program code generation and/or use.

What can be accomplished is an open-ended question.  I can only provide examples
of a few things that can be accomplished--which both Vesa and I have already
done.

> As an example, I understand that there is part of the Boost pp-lib
> which deals with easing the specification of repetitive constructs.
> That could be discussed, macros which solve repetitive generation
> problems can be discussed and explained, and the general user will
> now have an idea of what macros in the pp-lib help with  repetitive
> programming problems and how a given macro may be used in that way.

As I've said before, this categorization is already there:
docs/headers/repetition/*  I agree that this organization is not explicit, but
it is there for anybody who cares to look deeply at it.  There are examples of
how to use every interface macro in the library.  You seem to be asking me to
enumerate an open-ended set of possibilities.  How can I possibly do that?
Examples already exist, there are sections devoted entirely to the more complex
aspects of the library.  Given a general overview, what else can I do that would
please you?

> You may be right that the general user must first attempt to master
> the arcana of macro expansion and what does and does not work many
> macros attempt to expand inner macros in different situations in
> order to even use the pp-lib at all.

This is definitely part of it, but it has more to do with the vertical
dependencies associated with the implementation of each primitive in the
library.  How primitive X is implemented is a direct concern to primitive Y.
Expansion order is a significant factor as well.  Encapsulation is a huge issue
that causes use of the library to be closely tied to the implementation of the
library.  I have spent an excessive amount of time attempting to minimize these
dependancies.

> Or that likewise one must
> understand what can be done or not done given broken preprocessors in
> various implementations to use the pp-lib. I wouldn't say that is a
> failing of trying to use the pp-lib when the complexity goes up to a
> high level, but I would say if that were really the case, then the
> pp-lib has to attempt to create documentation which very patiently
> and in a well-organized way attempts to explain the above issues to
> the end-user.

This is the fundamental problem with what you propose.  Unlike virtually *every
other library in every other language*, programming with the preprocessor is one
large system all at once (for the most part).  Normal programming is broken into
smaller manageable pieces that interact via well-defined interfaces.  This is
simply not possible given the limitations of the preprocessor.  It is more like
manageable pieces that depend greatly on exactly how other manageable pieces are
_implemented_, not just their interfaces.  Any user-defined functionality or,
indeed, any actual use of the library inherently becomes part of the full
system.  That, combined with the lack of understanding of Cpp, is why the
learning curve is so steep.  It is not possible to construct documentation that
"eases" the user through the harsh realities of preprocessor metaprogramming.
The simple fact is that you have to get a "feel" for it by getting your hands
dirty.  Once you have that "feel," it becomes a powerful tool.  In fact, the
combination of the MPL and the pp-lib (especially the "strict" pp-lib that I'm
working on (which blows the CVS version out of the water)) is--by far--the most
powerful combination of tools for compile-time, generative, and reflective
metaprogramming that has ever existed in _any_ language (short of a full-fledged
intentional programming environment, of course. :) )

> Creating examples of already complicated uses of pp-lib
> macros, which the end-user can't possibly fathom yet, is not the way
> to explain these issues.

What examples do you refer to?  The examples that exist for each macro in the
reference section and the examples that exist in the topical sections are, for
the most part, as simple as can be.  You are saying that I should thoroughly
explain every macro used in examples at the point of the example, which is
unreasonable.

> It is possible that the Boost pp-lib is so intimately tied to template
> metaprogramming ideas that it is not really for the normal end-user
> who is just interested in knowing what it is all about and interested
> in seeing whether or not its functionality can be applied to other
> tasks. I will except this and, because I am not a template
> metaprogrammer in the Boost sense although I am very interested in
> what can be done in this area, it is simply not for me to understand
> or use. That's fine but I wonder if this is really the case with the
> Boost pp-lib. Most other Boost implementations do not require that
> sort of knowledge simply to be used by a C++ programming end-user. I
> understand the MPL does and perhaps the Boost pp-lib does too.

A level of competence with the preprocessor is required for all but the most
trivial (a.k.a. useless) application of the pp-lib.  This competence can be
gained through understanding of Cpp in detail and hands-on experience.  It is
inherently complex, which I can only obviate to a certain degree.  Preprocessor
metaprogramming _is_ closely linked to template metaprogramming, but it is not
tied to it.  It just happens that in many contexts the two conspire to provide a
solution to a given problem reasonably well.  As far as broken preprocessors are
concerned, because of the encapsulation issues that I mentioned earlier, there
are no hard and fast workarounds.  Also, I simply don't want to promote
workarounds for deficient preprocessors.  Rather I want to encourage vendors to
fix their preprocessors.

Again, I will write up an overview of the library this weekend and find out what
you think.

Regards,
Paul Mensonides