Re: [Boost-users] [Wave] extracting preprocessing grammar andcommentsfrom sourcefile

From: Hartmut Kaiser (hartmut.kaiser_at_[hidden])
Date: 2005-11-01 19:05:14

Andreas Sæbjørnsen wrote:

> I was hoping to avoid modifying the lexer itself, so I have
> been more reclined towards the approach of adding hooks to
> the library. I was thinking more in the direction of doing
> something similar to what  struct default_preprocessing_hooks
> (in preprocessing_hooks.hpp) does for macros, since the user
> can reimplement this for the instantiation of the template<>
> class context. Another option is to add this to template<>
> class context much similar to the way it is currently done
> for macros. What leads me to favour the struct
> default_preprocessing_hooks solution over modifying
> template<> class context is that it already handles similar
> problems  and you could also argue that these hooks does not
> fit in template<> class context. So my two options are:
>         - add more hooks to struct default_preprocessing_hooks
>         - add more member functions which work as hooks
> within template<> class context The hooks must be provided
> with all the necessary information for extracting the
> preprocessor grammar and evaluating the preprocessor
> conditionals.  What do you think is the best solution, its
> feasibility and how do you think it would fit into the wave
> preprocessor library?

I was thinking about to add new hooks to the preprocessing hooks template
from the very beginning. Sorry for not beeing consise enough.

For consistency reasons I'ld suggest to add a hook:

    template <typename TokenT>
    void
    found_directive(TokenT const& directive);

Where directive refers to the token containing the found pp directive.

> 	>               *evaluate the preprocessor conditionals
>
> 	What do you have in mind? Do you mean the
> (macro-)expanded conditional
> 	expression?
>
>  yes, so that I can extract the unexpanded preprocessor
> conditional expression and when this expression is
> (macro-)expanded if the result is positive or negative.
> For instance in the example
>         #define BAR
>         #ifdef BAR
>         int x;
>         #ifdef FOO
>         int y;
>         #endif
>         #endif
> I would be interested in extracting '#ifdef BAR' and also
> that it is evaluated as true. I would also be interested in
> extracting '#ifdef FOO' and that it is evaluated as false.

What about:

    template <typename ContainerT>
    void
    evaluated_conditional_expression(ContainerT const& expression,
        bool expression_value);

Where:
- expression contains the (not expanded) expression tokensequence and
- expression_value is the result of the evaluation of this expression.

> 	>               *after evaluating the preprocessor conditional,
> 	> extract the portion which was evaluated as false as a string
> 	>
> 	>                                      #define positive
> 	>                                       #ifdef int positive
> 	> #endif /*Extract this false part as string*/ int x; #endif
>
> 	Hmmm. This one is tough. The preprocessor is designed
> to skip this
> 	information, so I'll have to look at the code base how
> to best access the
> 	corresponding code fragments. Perhaps a special hook
> could be introduced to
> 	get called for every skipped token.
>
> That would be great! :) Do you think this can be done through
>  struct default_preprocessing_hooks?

    template <typename ContainerT>
    void
    skipped_token(TokenT const& token);

Where token is the skipped token. This hook will be called for each token
which gets skipped due to a false preproccessing condition.

> 	>               *I am also interested in the value and position
> 	> of unexpanded macros
>
> 	Undefined macros?
>
> I am only interested in defined macros. To be more specific I
> am interested in when the preprocessor recognises a macro.
> For each macro it is interesting to extract the value and
> position in the file. The macro can be found in two forms;
> the one before macro-expansion and the one after. Both forms
> are interesting. But this is from what I have seen already
> handled in struct default_preprocessing_hooks.
>                1:  #define FOO int x;
>                2:  FOO
> On line 2 in this example code the macro FOO is found. This
> macro can be expanded to 'int x', which to the preprocessor
> is equivalent to the unexpanded macro FOO found on line 1.

Yeah, I was already wondering how you might want to decide whether a
identifier actually is a 'undefined' macro :-P
This information already should be avalable through the existing
preproceesing hooks.

> 	>               *extracting the C/C++ statement or expression
> 	> that the unexpanded macro (and expanded) is
> 	>                 a part of.
>
> 	This conceptually isn't possible at the preprocessor
> level because it has no
> 	notion of a C++ statement/expression.
>
> I do not want to use Wave as a C/C++ parser, only to
> understand a subst of it's grammar. Let me corroborate for
> why I think it is doable and that the information necessary
> to do this is already easily available. What I was thinking was that
>       -since brackets ('{' and '}') and semicolon should is
> found within the tokens from the lexer, you should as far as
> I can see be able to fully define the grammar necessary to
> recognize what can be an expression or statment. For instance
> a variable declaration statement in C/C++ always ends with a ';'.
>       - a function definition statement has a basic block
> (body) which is always limited by the bracket ({...}).
>       -reference expression also tend to end with an ';',
> like for instance a function reference expression  " foo(); ".
> Therefore I would argue that since I do not think that you
> need an understanding of C/C++ syntax and only hopefully a
> fairly limited view of the C/C++ grammar (and the information
> for this is in the token-stream returned from the lexer) this
> should be doable. I think it can be a little bit difficult
> though, but I have to draw on your expertize here. Do you
> have any ideas for this?

I would not like to put any functionality into the library which does not
belong to its purpose: preprocessing C++. I'm pretty sure, that you'll be
able to build this on top of Wave.

> 	>               *extract the value and position of all C and
> 	> C++ comments
>
> 	This one is easy. Just enable the preserve comments
> mode and all the comment
> 	tokens will be part of the generated output token sequence.
>
> Great. :) What about making a hook for this within stuct
> default_preprocessing_hooks also?

Why? The preprocessing hooks are there to allow to access information not
available from the generated token stream itself, i.e. information about the
actual work inside the preprocessor. But the comments are available in the
generated token stream already. If I would add such a hook, somebody else
would like to have a special hook for line endings etc.

> It would be very interesting to do some work on this, and it
> would be useful to hear what you think about adding the
> additional hooks we have been talking about. Maybe these
> hooks should be better specified.

Do these new hooks satisfy your needs?

Regards Hartmut