RE: [boost] Re: Some remarks on PP docs

From: Paul Mensonides (pmenso57_at_[hidden])
Date: 2004-06-11 19:50:04

> -----Original Message-----
> From: boost-bounces_at_[hidden] 
> [mailto:boost-bounces_at_[hidden]] On Behalf Of David Abrahams

> > It is a reference tool intended to adequately support those 
> developers 
> > that are already familiar with the library--that is the 
> purpose of the 
> > layout.
> 
> Which begs the question of how anyone is supposed to get 
> familiar in the first place without personal help from you.

By actually reading the documentation.  All the necessary information is
there--though not all the information that could be there.  The documentation is
certainly not ideal, but every interface in the library is documented and fairly
easy to use if you actually read the documentation.  The primary thing that the
documentation lacks is a sort of categorical index with a broad overview of the
types of primitives that the library contains.

> Speaking as one of those who followed that route and is now 
> reasonably familiar, the layout isn't even very good for us.  
> The big blank space at the beginning and the lack of lines 
> around the frames are disconcerting, but more importantly, 
> the reference pane lumps every macro together at the same 
> level instead of categorizing them and progressively 
> revealing detail.  It would be a big help, for example, to 
> have a view that simply hid all of the names with _Z _R or _D 
> suffixes.

I would do that, except that scripting is disallowed by Boost policy and the
current documentation is hand written--that will change.  Regarding the
usability of the documentation as a reference...  From the initial starting
point, it takes all of two clicks to find the documentation for any primitive in
the library.

> > I'm not saying that the pp-lib docs look good--because they don't 
> > without the fonts fixed--but navigation is much easier and 
> much more 
> > direct.
> 
> It's pretty good, but the reference is much too "flat".  If 
> you really want to help the familiar programmer, you'll use 
> all of the hierarchical levels you have for a categorized 
> reference.  The parts that familiar programmers mostly don't 
> need to look at are currently sucking up all those convenient 
> navigational abilities.

What hierarchal levels are you referring to here?  Directory structure (i.e.
broad categorization)?  Doing such a thing would only add to the length of the
path from start to finish of a particular primitive.

> > Other documentation, such as Spirit's docs and Python's "Tutorial 
> > Introduction" are much more pleasing to the eye and much more 
> > professional.
> 
> I'd say they're very accessible but not particularly professional.
> They have a "chatty", almost dumbed-down (that's too 
> pejorative a word, but couldn't find another) feel about 
> them.  They're also very "flat".  Many people find the 
> "original iMac case" background disconcerting.

I'm not saying that they are top-of-the-line, just that they are "more
professional" looking than the standard HTML 1.0 docs.  Personally, I dislike
anything that ties the format to a particular look--mainly because what I'd
prefer is likely to be different than many others--but also because of the
separation of content and presentation.  However, without scripting or other
more advanced features, we'd have to do that in advance and publish a specific
one.  That is still better, but only from the documentation-writer's point of
view.  Further, XML is a terribly wordy and boilerplate-ridden source format.
Minor formatting can be changed through stylesheets, but major formatting
changes require a new XSL transformation entirely.  None of these problems are
unsolveable, but they get worse and worse the more general purpose the schema is
for documentation of a variety of different forms.

> >> I didn't get that either.  It's a shame, because there's a lot of 
> >> code that would benefit from the PP lib that almost 
> certainly isn't 
> >> because of the difficult learning curve.
> >
> > The difficult learning curve is never going to go away.  
> 
> Mostly because you believe it can't.

No, mostly because of the nature of the subject.  I'm not saying that it can't
be easier than it is.  I'm saying that it can't be simple.

> > It is an alien environment that is contrary to most forms of 
> > programming.  That takes time to become familiar with.
> 
> That's absolutely true, but most people don't need to get far 
> enough along on the curve to get to the part where it really 
> *has* to be difficult.

The primitives of the library, barring a relative few, are easy to use, and
their documentation is easy to follow.  That is not where the learning curve is
steep.  Rather it is how to apply the primitives available to solve problems.
That requires lateral thought--and that is what makes the learning curve steep.
There is no one-to-one mapping between some particular primitive and some
particular problem to be solved.  "Lateral thought" is, for example, when you
have A and B from which you conceive the abstract possibility of C.  SFINAE, and
enable_if specifically, is a good example.  It takes lateral thought to go from
type deduction failure to manual overload/specialization selection.  That kind
of application is certainly not always necessary to use the pp-lib for trivial
tasks, but it is necessary to use the pp-lib well in non-trivial tasks.  This is
no different than C++ or any other programming language, which has a large array
of features whose intersection makes C++ complex (and also interesting).  The
difference is that a great deal of literature exists on programming paradigms in
computer science and various languages, but none exist for the language that is
the preprocessor--and by extension, preprocessor libraries--which is
fundamentally different in many ways (even procedural abstraction doesn't
exist).

> I didn't get started with C++ by 
> studying the overload resolution rules, did you?  I still 
> don't know many of the subtleties, and yet I get plenty of 
> useful work done; I even write and call overloaded functions 
> on a daily basis.

Yes, but you also didn't start writing C++ code by reading the documentation for
template metaprogramming library X.  You likely started with some book about C++
itself.

> >> Good documentation would go a long way towards making it 
> more usable.
> >
> > More usable for what?  
> 
> Weren't you the one who was frustrated by people asking "what 
> can the PP lib be used for?" ;-)

That is precisely the point of the rhetorical question the following exposition.

> > I ask this simply to determine perspective.  The documentation is 
> > never going to tell you what you can and cannot do with the 
> > preprocessor or what you should and should not do with the 
> > preprocessor--those are all unbounded sets.
> 
> Paul, your ability to move from the specific to the general 
> is awesome; it's part of what makes you such a great 
> programmer.  Your unwillingness to go in the other direction 
> is a liability, though, at least for your users.

I am not at all unwilling to go the other direction.  I am simply stating a
fact: there is a many-to-many relationship between the primitives of the library
and the problems that they combine to solve.  I cannot answer the question,
"What types of things is the pp-lib good for?", because the answer is an
unbounded set.  I can present only a finite set of such solutions as examples
and present the building blocks from which any given solution in the unbounded
set can be derived as needed.

> The PP lib is really only very well-suited to algorithmic and 
> pattern-based C/C++ code generation.

Nevertheless that is still all-encompassing.  In order to do X, you often have
to do Y which is in many cases outside of the particular field of X.

> You probably could 
> build a program that analyzes english sentences in PP 
> language, but other languages are much better suited to AI research.

Agreed.  However, the uses of the pp-lib within Boost itself barely touch the
potential--it just represents a subset of problems which developers happened to
come in contact with.

> > The most that the documentation can do is tell you what the 
> _library_ 
> > can do and give a few examples.
> 
> It can describe the kinds of jobs the library is well-suited to.

The library is well-suited to these kinds of jobs: repetition, looping,
recursion, and control flow.  It isn't more suited to generating arity
expansions, which is the typical Boost usage, than many other possible uses
within those extremely broad "kinds of jobs".

> > Take ENUM_PARAMS for example, an extremely common use of this 
> > primitive is to generate stuff like "class T0, class T1, class T2".
> > However, the library doesn't have a single primitive that generates 
> > that string of preprocessing tokens--it only has the general form.
> > It could just as easily generate "0.1, 0.2, 0.3".  
> 
> And you think that's hard to explain in a way that's simple, 
> yet captures the generality of ENUM_PARAMS?

No, it isn't hard to explain ENUM_PARAMS.  It is hard to explain when and where
one should use ENUM_PARAMS except for a few very specific situations.  Of all
the constructs in the library, that one is also the easiest to explain as a
mapping between primitive and solution because it is the closest to the
underlying language.

> > The point is that the output is a function of the combination of 
> > primitives, user-defined macros, and input.  The library takes the 
> > role of a programming language.  As such, it is never going 
> to answer 
> > questions like "what things can I do with 'if'?"  and "what 
> things can 
> > I do with 'while'?".
> 
> I don't know why you obsess over questions like that.  I 
> don't think anyone expects anything other than a reasonable 
> answer to them.  For example, "BOOST_PP_IF selects between 
> two sequences of preprocessing tokens based on an 
> integral-valued argument" (roughly speaking).

And this documentation is already there.

> > The library really is that low-level.  ENUM_PARAMS is about 
> the most 
> > underlying-language-oriented primitive in the entire 
> library.  There 
> > is definitely room for improvement, but I suspect that no 
> matter how 
> > much I improve the docs and no matter how many examples that I add, 
> > I'm always going to hear the same old argument--the 
> learning curve is 
> > too steep.
> 
> As long as you insist that it's impossible to (or people 
> shouldn't) use the library without an in-depth understanding 
> of the fundamental operation of the PP, that will be a 
> self-fulfilling prophecy.

I didn't say that an in-depth understanding is required.  I said only that the
learning curve is steep because it is a different programming language with a
different programming model than any existing language--and because that
programming model doesn't have several decades worth of research, experience,
and literature.

> > That argument is based on the assumption that documentation 
> (etc.) is 
> > enough to reduce every learning curve to a gentle slope--which is 
> > naive.  The library is hard to learn
> 
> It doesn't matter whether the library is hard to learn.  It 
> matters whether it's hard to learn _to use_.  Right now, 
> that's the case.

I am not talking about how the library itself is implemented; I'm talking about
how the library is applied.

> Most people learn much more effectively incrementally, and in 
> layers.  The current docs, for all intents and purposes, dump 
> the whole library on you at once.

That is true, but that is also because the library is not incremental.  It is a
collection of relatively low-level primitives that you can combine to do
interesting and worthwhile things.  Yes, these could be categorized in a more
explicit fashion, but that is nothing compared to the combinatorial explosion of
documentation resulting from attempting to document the abstract perception of
element C (from above).  In nearly every case, single library primitives are not
particularly useful without others used in concert to achieve some effect.

> > because the library is different than anything that most 
> programmers 
> > have been exposed to and because the many of the techniques 
> and idioms 
> > used are different than those used in every other language and in 
> > computer science in general.  It isn't simple, and it can't 
> be made to 
> > be simple.
> 
> Sure the whole picture can't be made simple, but enough of it 
> can be made simple that people could get a whole lot more 
> useful work done without handholding from you.

I'm not talking about the whole picture.  I'm talking about how to apply the
pp-lib, not how it is implemented and especially not about preprocessor
metaprogramming in general.  Take something as simple as IF, it isn't really
good for anything useful unless it is used in some larger practical example.
Sure I can document IF as an isolated entity and provide simplistic examples of
its use, but that doesn't really show how or why IF should be used or why it's
actually useful at all.  Sure I can provide larger practical examples, but then
the learning curve dramatically steepens because much more information is
required to be processed at once (i.e. the use of many more primitives and
techniques as well as semi-common fields like generic programming) in order to
understand why the example is better than various other alternatives.

> With some 
> sense of mastery under their belts on that basis, they would 
> be much more inclined to delve deeper, not least because it 
> would seem easier.

Given initiative, one can learn how to use the preprocessor library.  How many
people that have complained about the documentation have actually read through
the entire documentation and tried the examples, etc.?  The goal of learning how
to use the pp-lib is eminently achievable, but you have to invest the time
required to be reasonably comfortable a different language and a different way
of looking at source code.

Regards,
Paul Mensonides