Re: [Boost-users] [WAVE] position of tokens

From: Andreas Sæbjørnsen (andreas.saebjoernsen_at_[hidden])
Date: 2006-11-09 15:56:55

> I changed the real_positions example to use a new token type carrying both,
> the original and the corrected positions. This makes it a bit more tricky,
> though, because now you have to explicitely instantiate some of the internal
> Wave library template types for this new token type. These explicit
> instantiations are normally done in the Wave library code (for the default
> token type).

I think this probably is the optimal solution and it is not difficult
to instantiate the templates in my library although I expect it to
slow down compilation considerably (?).

>
> Note: the new token type is essentially a copy of the default token type
> which has added the second position instance. For simplicity reasons I
> removed the class allocators, though. You'll have to re-add these, if
> needed. Please look at the file cpp_lex_token.hpp for reference).
>

In which instances would I need to use these class allocators? What
are they used for in Wave with the standard token type?

> > Yes. It was very easy to work with your code so that should
> > be no problem. Are the real_positions set before or after the
> > tokens are send to the preprocessing hooks?
>
> The corrected position is set inside the generated_token() pp hook. IIUC
> correctly you're asking, whether the corrected token position will be
> available in other hooks as well. The answer is no. The generated_token pp
> hook is call at the very last point before the preprocessed token gets
> returned to the calling application. I see no other way to do this, because
> only at this point I know everything about the final token sequence. Any
> ideas?

An idea would be to allow calling of generated_token() from one of the
other hooks. For instance, if you in rescanned_macro() know when the
first macro has been fully texpanded then you can call
generated_token() at that point on every token provided to
rescanned_macro(). The reason why I say first macro is that a macro in
it's formal definition can call another macro, so first macro calls
second macro etc.

The suggestion I made above can probably also be considered an
optimization (?) because you could also imagine recomputing the
correct positions on every call to rescanned_macro.
Do you see any reason for this not working?

Thanks
Andreas
> >
> > > > On 11/7/06, Hartmut Kaiser <hartmut.kaiser_at_[hidden]> wrote:
> > > > >
> > > > > Andreas,
> > > > >
> > > > > > I need some advice regarding Wave. What I want to do is
> > > > to operate
> > > > > > with Wave on a token stream and extract the token stream
> > > > > > representing macro calls after expansion. The twist is
> > > > that I want
> > > > > > the token stream after expansion to have the positions
> > > > that another
> > > > > > preprocessor would see on the already expanded output
> > from Wave.
> > > > > > E.g:
> > > > > > #define MACRO_CALL int x;
> > > > > > int main(){
> > > > > >      MACRO
> > > > > > };
> > > > > >
> > > > > > Exands into
> > > > > > #line 2
> > > > > > int main(){
> > > > > >      int x;
> > > > > > };
> > > > > >
> > > > > > Where 'int x' should have a position of line 3
> > instead of line 1
> > > > > > after expansion. I also want the column number to
> > > > represent what we
> > > > > > see on line 3 instead of what we see on line 1. Is
> > this possible ?
> > > > >
> > > > > Ok, I've added a new pp hook (what else? :-P) to Wave:
> > > > >
> > > > >
> > > >
> > ////////////////////////////////////////////////////////////////////
> > > > //
> > > > > /////
> > > > > //
> > > > > //  The function 'generated_token' will be called by
> > the library
> > > > > whenever a //  token is about to be returned from the library.
> > > > > //
> > > > > //  The parameter 'ctx' is a reference to the context
> > > > object used for
> > > > > //  instantiating the preprocessing iterators by the user.
> > > > > //
> > > > > //  The parameter 't' is the token about to be returned
> > > > from the library.
> > > > > //  This function may alter the token, but in this case it
> > > > must be //
> > > > > implemented with a corresponding signature:
> > > > > //
> > > > > //      Token const&
> > > > > //      generated_token(Context const& ctx, Token& t);
> > > > > //
> > > > > //  which makes it possible to modify the token in place.
> > > > > //
> > > > > //  The default behavior is to return the token passed as the
> > > > > parameter //  without modification.
> > > > > //
> > > > >
> > > > //////////////////////////////////////////////////////////////
> > > > /////////////
> > > > >     template <typename Context, typename Token>
> > > > >     Token const&
> > > > >     generated_token(Context const& ctx, Token const& t)
> > > > >     { return t; }
> > > > >
> > > > > This should help solving your problem.
> > > > >
> > > > > To show how this hook function may help you I added a
> > new sample
> > > > > 'real_positions' to Wave demonstrating its use. Everything
> > > > is in the
> > > > > Boost CVS::HEAD.
> > > > >
> > > > > HTH
> > > > > Regards Hartmut
> > > > >
> > > > > _______________________________________________
> > > > > Boost-users mailing list
> > > > > Boost-users_at_[hidden]
> > > > > http://lists.boost.org/mailman/listinfo.cgi/boost-users
> > > > >
> > > > _______________________________________________
> > > > Boost-users mailing list
> > > > Boost-users_at_[hidden]
> > > > http://lists.boost.org/mailman/listinfo.cgi/boost-users
> > >
> > > _______________________________________________
> > > Boost-users mailing list
> > > Boost-users_at_[hidden]
> > > http://lists.boost.org/mailman/listinfo.cgi/boost-users
> > >
> > _______________________________________________
> > Boost-users mailing list
> > Boost-users_at_[hidden]
> > http://lists.boost.org/mailman/listinfo.cgi/boost-users
>
> _______________________________________________
> Boost-users mailing list
> Boost-users_at_[hidden]
> http://lists.boost.org/mailman/listinfo.cgi/boost-users
>