Re: [boost] Preprocessor: Nested loops

From: Paul Mensonides (pmenso57_at_[hidden])
Date: 2006-02-16 08:29:48

> -----Original Message-----
> From: boost-bounces_at_[hidden] 
> [mailto:boost-bounces_at_[hidden]] On Behalf Of Oliver Kullmann

> > What is a "plain example"?
> >
> 
> An example which shows one thing, without any additional 
> fuss, so that it immediately clear what the example is doing, 
> and how it is achieved.

Those are the ones that you (and I) call contrived.  There are many of those.

> If I enter the current documentation, then first I get to the 
> short introduction page with a link to the chapter from the 
> mpl book, which is based on earlier chapter in the book, and 
> thus cannot be used as an introduction.

It isn't really based on an earlier chapter.

> So then I continue to Topics (the next item on the list), 
> where the natural choice seems to look for "Motivation": But 
> here again there is no plain example, but we need to know at 
> this point what the "is_function<> metafunction in Boost" is 
> (no attempt is made to explain this thing).
> So the solution presented down the page doesn't motivate 
> anything for anybody who is not familiar with all of Boost 
> (it is not even said where we could find this function in 
> Boost, i.e., to what sub-library it belongs).

Where does the documentation refer to an "is_function<> metafunction in Boost"?
(The documentation uses some similar things as examples.)  Incidentally, I
didn't write the first three of the documents in the "Topics" section, but I did
write the rest of them.

> Such examples I call "non-plain". A plain example would 
> consider for example the problem of how to create an 
> enumeration out of a variable list of identifiers. The point 
> is here that the example is *simple*, that is, atomic --- not 
> much reference to anything else is needed.

And I'd call that a bad example that promotes a bad use of the library.  Taking
this one apart, you don't gain anything by defining a simple enum like this.
Instead, you might gain something if you define not only the enum, but also
surrounding code that makes the enum more useful than a built in enum (e.g. I/O
support, etc.).  But when you do that, the example is suddenly much more
complicated.  As I said before, this kind of thing does not boil down to simple
use cases--which is exactly what this kind of example is.  Instead, when the
abstractions provided are brought all the way down to concrete use cases, those
use cases are either bad uses or are complex.  This is almost invariably true.
If it wasn't, you wouldn't need preprocessor metaprogramming to do it.

> > > and thus I skipped that and went to Reference, where I found 
> > > BOOST_PP_SEQ_FOR_EACH, making some still mysterious remarks about 
> > > "The next available BOOST_PP_FOR repetition."
> > > etc. But at various places it somehow says that "reentry problems"
> > > whatever this means are solved now.
> > 
> > There is an article in the topics section about this 
> (titled "Reentrancy").
> >
> 
> How does the reader of the documentation get there? After I 
> found that the Motivation does not motivate much for me, I 
> went on to "Known problems" --- alright, I understand that, 
> but it doesn't give me much. So well, going to techniques : 
> Again the very first example doesn't say what it is supposed 
> to do, and so it continues; all examples are part of some 
> discussion within an expert round of users (who don't bother 
> to even spell out the problems they are discussing).

Expert users are the primary targets of the documentation.  This is a generative
_metaprogramming_ library.  There are all kinds of things that you can do with
metaprogramming--one of them is write code that is total crap.  The target of
the library is advanced programmers, because only advanced programmers have
accumulated the design  experience necessary to evaluate the tradeoffs involved.
Furthermore, many Boost authors can readily identify with the underlying problem
that something like 'is_function' takes in stride (and what it really is an
example of).  That's dealing with variable arity--a very common area in which
the library is used.

> So let's skip techniques. Next comes "Incompatibilities". It 
> speaks about the previous Boost release 1.28, so it seems not 
> of relevance anymore (especially since I'm a new user).

No, that one isn't relevant to you.

> So let's skip this, going to "reentrancy". From the headline 
> it's not clear what could be meant by it (if you don't know 
> it). Alright, the first few paragraphs there name the problem.
> But actually not explicitely, it doesn't say that C or C++ in 
> version soandso has this "feature", but it speaks of "the 
> preprocessor", which might mean a specific compiler, some 
> special features or this very library itself.

The preprocessor is defined by the standard, and that is what the documentation
is referring to.  Many real world preprocessors approximate it closely, and some
are horribly broken.

> It also speaks 
> of recursion, which in the usual sense of the word (that is, 
> in mathematics) is not what is really meant.

I'm not sure I follow this.  "Recursion", as I'm using the term in the
documentation and as it is commonly used in computer science, is the ability for
an interface to use itself.

> I want to make the point here, that as a documentation author 
> one has similar responsibilities to a car driver: One must 
> anticipate problems. Readers always have a certain special 
> angle how the view it, and one must add many hints and 
> redundancies to help them becoming aware of the incongruence 
> of their point of view with the authors point of view.

I don't disagree with that, but I also can't accommodate every possible point of
view.

> My 
> main assumption was, that C and C++ respects the programmer, 
> and thus does not disallow potentially "dangerous 
> constructions". So in my mind I glanced over the problem with 
> "recursion", and basically thought that this is no problem 
> for C and C++ (the whole process might go wrong, but we are 
> in control).
> Coming from this point of view, I then see "recursion", guess 
> they mean something different, and moreover I have already 
> seen somewhere that "reentrancy" (whatever this means) is 
> solved by the new version of the library anyway,

The documentation doesn't say that *anywhere*.

> so I don't 
> need to bother (not to forget that when I come to this point 
> in the documentation, then I already "learned" that 
> apparently most parts of the documentation have to be skipped).

Or read with an attempt to understand what they are saying...

> > > Only when looking it up
> > > in the rationale of C99 I found a clear statement about the issue 
> > > (while the C++ standard somehow supposes you know already the 
> > > issue).
> > 
> > It is pretty clear in both standards (6.10.3.4/2 in C and 
> 16.3.4/2 in C++).
> >
> 
> I'm not aware of *any* book on C or C++ discussing this 
> issue. I have quite a collection of them, read (or have read) 
> CUJ, CVu and Overload, but the preprocessor seems to be a tabu.
> This I think one should keep in mind for the documentation.

So, what you're saying is: 1) there is no 20+ years of existing literature as
there is for C and C++, so 2) this library should provide the equivalent of 20+
years of literature.  The bottom line is that the responsibility of the
documentation stops after documenting itself.  Anything beyond that is a
freebie--the documentation is going above an beyond the call, so-to-speak.  I'm
not against that, but I'm hearing nothing but "everything is poorly done".

> Alright, leaves the standard(s). I worked through all 
> examples there, but none of them shows the "recursion 
> problem" (at least in the C++ standard). Sure, finally one 
> gets to the point where you understand what they are talking 
> about, but it takes (too) long. You start reading this chapter

I agree that the standard is not elucidating.  I disagree that it is my
_responsibility_ (via the documentation) to elucidate it, regardless of how
useful it might be to do so.  Frankly, the lack of recursion is an fundamental
aspect of the way macro expansion works.  If you don't know that (and I don't
mean all the details down to the subatomic level), you probably aren't *ready*
to use this library.

> And so on. Among all of that you have to find the right few sentences.

Yep.  That's because it is a technical specification, not an explanation.  Yes,
it is difficult to follow and get all the details.  Yes, the elucidation that
you want would be useful.  No, it is _not_ my responsibility to do so--even
though I'm already in the process of doing exactly that for macro expansion.  On
top of all this, I'm doing it in my spare time for free.  Just like I'm spending
hours on this conversation (and putting off other things that I'm working on).

> First, text read by humans is different from text read by 
> computers. Due to the inherent imprecision of natural 
> language, and the necessity for the reader to anticipate 
> meaning (a well-know psychological fact that without 
> anticipation no understanding), the reader constantly goes 
> into wrong directions, and only redundancies (saying 
> important things at many places (with different words)) can help him.

Technical documentation needs to be concise.  I shouldn't have to say the same
thing twice.  If the reader doesn't understand something later, they can go back
and re-read it.

> And second, if the documentation would be organised like 
> software, this might actually help:
> Every file(!) (not just every translation unit) is supposed 
> to include for example <algorithm> again and again --- 
> accordingly one would at least expect at every reference for 
> some construction (corresponding to a source code file) a 
> *link* to the relevant general sources here ("including"
> that what needs to be known to understand the reference).

I agree with this--as long as it is about library-defined issues.  E.g. I don't
have a problem with providing cross-references to material on algorithm states.
I do have a problem providing cross-references to basic preprocessor
functionality every time it might possibly be relevant (which is just about
everywhere).

> > > Second of all it seems the current facilities in the 
> library should 
> > > be enhanced (though I don't understand how FOR and FOR_r 
> is supposed 
> > > to be used): BOOST_PP_SEQ_FOR_EACH is such a natural and 
> easy thing, 
> > > the use of it should be promoted, while FOR and its 
> derivates look 
> > > ugly.
> > 
> > Hmm.  FOR is far more general.  SEQ_FOR_EACH can be 
> implemented with 
> > FOR, but not vice versa.
> >
> 
> sure --- in other words, FOR_EACH is easier to use!

Sure, as are a whole lot of other things that are just as easy to use as
FOR_EACH.  IOW, it is nothing special; it is one of many, etc., etc..

> > It will only break if SEQ_FOR_EACH or LIST_FOR_EACH arbitrarily 
> > decides to use the other--which won't happen.  I design the 
> algorithms 
> > in the library, and I'm well aware of vertical dependencies 
> and their implications.
> >
> 
> I was referring to the documentation, which seems not to 
> speak about this issue.

The documentation says that certain constructs are reentrant.  It doesn't say
for every non-reentrant macro that it is non-rentrant--because that is the
status quo, the "normal" situation which is not defined by the library and thus
is not the responsibility of the library to document.

> > > According to what I understand from the documentation, 
> the natural 
> > > solution, offering for example BOOST_PP_SEQ_FOR_EACH_2, 
> which uses 
> > > macros guaranteed not to occur in BOOST_PP_SEQ_FOR_EACH, has been 
> > > deprecated in favour of a more complicated solution.
> > 
> > You don't understand the implications of what you're saying.
> 
> sure

I'm not saying that to be derogatory, but I am saying that out of frustration
because you immediately started pronouncing what should be and what I should do
without knowing any of the implications of your pronouncements nor any of the
rationale behind the design of the library or what is and isn't in the
documentation.  You didn't ask *why* something is the way it is.  Instead, you
ran into problems, and then said that everything should be better (so that you
wouldn't have run into problems).  I'm also not belittling the problems that you
ran into.  Preprocessor metaprogramming can be tricky, is very foreign to
typical programming models, and has a steep learning curve.  Sure, the
documentation isn't perfect, but sometimes learning curves are inherently steep.

> > In order to do
> > that, I'd need to define hundreds of macros to implement 
> SEQ_FOR_EACH 
> > alone.  It isn't as simple as just replicating the 
> interface macro a 
> > few times.  Instead, I'd have to reimplement the entire 
> algorithm from 
> > scratch--intentionally avoiding the use of any other 
> higher-order algorithm.
> >
> 
> the whole thing looks quite complicated; why not adding a FAQ 
> page to the documentation (I was looking for one, but didn't 
> find one) answering for example my above request in this way? 
> I think it would have helped me.

Because I don't want to implement the library in the documentation.  These are
implementation details and if I introduced those into the documentation, the
documentation would be orders of magnitude more complex.

> > I'm not sure what you're referring to by "more complicated 
> solution".  
> > If you specify what kind of thing your referring to, I'll expound.
> >
> 
> The documentation says
> 
> In particular, the library must provide reentrance for 
> BOOST_PP_FOR, BOOST_PP_REPEAT, and BOOST_PP_WHILE.  There are 
> two mechanisms that are used to accomplish this:  state 
> parameters (like the above concatenation example) and 
> automatic recursion.
> 
> Thus it seems only for these loops you have a special 
> mechanism for reentrancy, and this special, "generic" mechanism"
> is what I mean with "more complicated" : for the user it's 
> easy to use BOOST_PP_SEQ_FOR_EACH and 
> BOOST_PP_SEQ_FOR_EACH_2, but it's complicated to use the 
> generic method.

Except that you're wrong.  You run into massive dependency nightmares if you do
that.  It isn't simpler--it creates problems that are very difficult for users
to track down.

> > The documentation has many examples, but you call them either 
> > complicated or contrived.
> 
> Let's go to the example section:
> 
> First we have "array_arithmetic.c": It says "This example 
> implements over 2200 functions".
> Sounds impressive, but is not an example (sounds more like an 
> extension of the library).

No, it's an example.  The library is a code generator, not a library of
generated code.

> Among those six examples only "catch_builtin.cpp" seems to be 
> close to an example (which is something you look at and you 
> understand).

I disagree with your definition of an example.  Any code that uses an interface
is an example of using that interface.  Whether or not it is a good example is
another issue.  However, the library includes many very simple examples--such as
the example for SEQ_FOR_EACH:

#define SEQ (w)(x)(y)(z)
#define MACRO(r, data, elem) BOOST_PP_CAT(elem, data)

BOOST_PP_SEQ_FOR_EACH(MACRO, _, SEQ) // w_ x_ y_ z_

That is a simple example that clearly shows what SEQ_FOR_EACH does without
focusing on an overarching (complicated) problem to be solved.  The library is
littered with such examples.

> In the reference section there are actually simple examples, 
> but as far as I can see always only one example: I believe a 
> systematical declination of the elementary cases would be 
> very helpful (always staying simple, just handling simple 
> sequences, but with example
> 5 to BOOST_PP_SEQ_FOR_EACH for example on would show a nested 
> loop ("without added meaning", that is, no template stuff etc.)).

Why would I show a nested loop in an example for SEQ_FOR_EACH when SEQ_FOR_EACH
cannot be called from the repeated macro invoked by SEQ_FOR_EACH?

> > What exactly to you want?  This kind of stuff doesn't boil down to 
> > _simple_ use cases.
> 
> My understanding is that use cases are not very helpful. What 
> we need is *understanding*.

Yes.

> ??? What does this want to say to us? What are "graphs" etc. 
> ??? But shouldn't one know all these things!! And it's a very 
> nice fundamental property of graphs, just *using* addition in 
> two ways.

The difference between this mock example and the library's examples is
significant.  The templates (or anything else) that appear in the examples
aren't what the examples are about.  They are *output* of the example--not part
of the functionality of the example.

> sum_{i=1}^n i = n (n+1) /2.
> 
> Hope you get what I mean.

I get what you mean, but I think the simple examples are already there.  I just
don't think you invested much effort.  It basically seems that if you saw
"template", you just immediately ignored the example.

> >  Rather, typical use cases are parts of overarching designs 
> that are 
> > complex--which is precisely the reason that preprocessor 
> > metaprogramming enters the picture at all.
> 
> As I said, I don't want use cases, I just want to use the 
> preprocessing library (for my own purposes).

Okay, but your example of using the library to generate an enum is a use-case.
Sure, it's a simple one, but it is only simple because it is naive to the point
of being destructive.

> > Furthermore, I have a responsibility
> > (as does any library author) not to promote naive or poor 
> uses of the 
> > library--which is almost always the case with examples that 
> implement 
> > something simple.
> 
> So the documentation is the guard, meant to be as a kind of 
> torture, and only those worthy souls which get through shall 
> use the library?

This actually makes me smile.

No, the documentation is not meant to be that, but the learning curve is
inherently steep, and such learning requires the learner to expend a certain
amount of effort--moreso than when learning something simpler.

What I said was true.  I have a responsibility not to promote naive or poor uses
of the library.  The user that is learning the library has not yet attained the
degree of aptitude required to separate the way something is achieved (the point
of the example) from what is being achieved (the medium of the example).

> Speaking for myself, I never ever found any interest in those 
> "real world scenario's"
> (for this library and others), since their "real world" is not mine.
> Of course, other might disagree, and having a healthy mix of 
> simple and more complicated examples sounds like a good thing.

It is a good thing.  What you seem to be suggesting, as far as examples are
concerned, is somewhere in the middle.  Something that I would call a trivial
use case.  But for the reasons that I already mentioned, such examples can be
dangerous.  Thus, I tried to either make them full-fledged and concrete or
utterly simple and contrived.  (There were a few times that I got bored of
writing ultra-simple examples, so I made them a little more elaborate, but those
are almost always in the interfaces targeted towards more advanced users.)

> > The library 1) shouldn't have to announce this problem
> 
> I meant the documentation

As did I.

> > and 2) already does
> > announce this problem--it just doesn't do it hundreds of times in 
> > hundreds of different places.  If you are writing code in 
> the language 
> > defined by the preprocessor, you should *at least* know the 
> basics of 
> > that language.  The library is not the language, it is merely a 
> > systematic set of reusable constructs written in that 
> language.  The 
> > documentation's only responsibility is to document itself.
> >
> 
> yes, but neither you are an angel (so that you could give a 
> precise definition of the whole
> thing) nor am I (so that I could read it); instead we are 
> human beings nd mst fnd r wy thrgh ntcptn (there are nicer 
> examples; but hopefully this works a bit:
> just looking at it I hope you thought huh, but knowing that I 
> left out the consonants it shouldn't be too hard; I have seen
               ^^^^^^^^^^ vowels?

> a very nice series of examples in this style, where you were 
> guided to read more and more strange stuff, and at the end 
> you were presented with a plain English sentence --- and you 
> couldn't read it anymore!).

You're right that documentation has to be more than just a formal definition.
However, the point remains that it isn't the job of the documentation of a
library written in language X to document language X.

> >  Regarding examples in the
> > documentation, I don't think there is a way to please you
> 
> perhaps there are ways: I believe actually in small steps, 
> massaging something reasonable here and there so that it gets 
> better. If some of the above thoughts would lead to some 
> additions here and there (a few remarks, hints) in the 
> documentation, that would mean progress.

Out of curiousity--do you believe there are certain things that have inherently
steep learning curves?  What you're suggesting here is basically that I flatten
the learning curve by gradually introducing complexity.  My argument is that
there are times that complexity cannot be avoided or compartmentalized into
small, easily understood, chunks.  There are certain steps along the road that
are transactional--all or nothing.

> > Regarding the technical implications:

> I can understand the general point of view. But this general 
> point of view might not necessarily be the point of a user 
> (who in general will just occasionally use the library).
> 
> Perhaps it might be a good idea to add the above explanation 
> somewhere to the documentation?

IOW, add implementation details to the documentation?  As far as the reentry
stuff is concerned, all that is necessary for users is to know is that macros
cannot be used inside themselves (i.e. common knowledge).  The places where that
rule is broken (meaning that the library makes it look like it is breaking the
rule) are explicitly noted.  Users need not ever deal with state
parameters--they only need to know that macros will not expand inside
themselves.  Thus, the macro passed to (e.g.) SEQ_FOR_EACH, that is invoked by
SEQ_FOR_EACH, cannot use SEQ_FOR_EACH.  Otherwise, the library makes it work
automatically.  You aren't required to use SEQ_FOR_EACH_R--or any interface that
requires you to mess around with an algorithm state.  Instead, you can ignore it
completely.

> So if I understand it correctly, you currently have three 
> independent looping constructs BOOST_PP_FOR, BOOST_PP_REPEAT, 
> BOOST_PP_WHILE, while all other loops are implemented in 
> terms of these?

There are actually more than that, but the library makes it appear that there
are only a few algorithmic states (e.g. 'r' for FOR, 'z' for REPEAT, and 'd' for
WHILE), and therefore that there are only a few algorithms implemented this way.

It used to be the case that, because SEQ_FOR_EACH was implemented in terms of
FOR, you couldn't use SEQ_FOR_EACH inside FOR.  You *had* to use SEQ_FOR_EACH_R.
I removed that kind of limitation.  You cannot use SEQ_FOR_EACH inside itself,
but you need not deal with the algorithm states (like 'R') at all if you don't
want to.  This makes the library a great deal easier to use.

> Then it would make sense to concentrate on the documentation 
> of these three macros, giving more examples (showing also how 
> to simulate nesting), and emphasising for e.g. FOR_EACH its 
> special character.

It isn't FOR_EACH that is special, it is the ones that are explicitly designed
to be reentrant that are special.

> As far as I can see from the material provided on more 
> powerful pp-libraries, the Boost pp-library won't be further 
> developed, but at some time replaced?

Not necessarily.  However, it is near what I can do with it either because of
portability issues or efficiency concerns with various preprocessors.  I have
tried adding various things to it, but it is extremely difficult to keep the
library stable on some very popular compilers.

Things are improving.  As far as broken preprocessors are concerned, there are
only a few remaining roadblocks--the most significant being VC++.  As far as
efficiency is concerned, the major roadblock has always been EDG-based
preprocessors (notoriously slow).  I haven't seen proof of this, but
*supposedly* the EDG preprocessor has been modified to use memory much more
efficiently during macro expansion, and the previous way that it was using
memory was *supposedly* the source of the major slowdown.  (This is in an EDG
version that is not yet available in any other compiler.)

The speed issue is not as big an issue as the utter broken-ness of VC++ in
particular.  When and if VC++ and the other two broken preprocessors, Sun and
IBM, fix their preprocessors, the library won't be replaced so much as heavily
refactored.  Some other refactoring is due anyway in preparation for variadics.
As an immediate example, SEQ_FOR_EACH's interface needs to change from:

SEQ_FOR_EACH(macro, data, seq)

to:

SEQ_FOR_EACH(macro, seq, data)

in preparation for it becoming:

SEQ_FOR_EACH(macro, seq, ...)

Of course, when and if that happens--which depends on the vendors--a lot of the
limitations that you've run into will be gone.  SEQ_FOR_EACH would be just as
reentrant as anything else.

Regards,
Paul Mensonides