Re: [Boost-users] [WAVE] position of tokens

From: Hartmut Kaiser (hartmut.kaiser_at_[hidden])
Date: 2006-11-09 09:47:15

Andreas Sæbjørnsen wrote:

> > >     * the position of a token in the original source code is 
> > > gone(default in standard Wave)
> >
> > What do you mean by that?
> 
> What I need is both the original position and the position 
> from the real_positions example available on a token.  This 
> makes it possible to locate a token using the real_positions 
> (which is equivalent to positions on output from a 
> preprocessor) and then get the original source code position 
> from that token. Do you see any other way of achieving this?

I changed the real_positions example to use a new token type carrying both,
the original and the corrected positions. This makes it a bit more tricky,
though, because now you have to explicitely instantiate some of the internal
Wave library template types for this new token type. These explicit
instantiations are normally done in the Wave library code (for the default
token type).

Note: the new token type is essentially a copy of the default token type
which has added the second position instance. For simplicity reasons I
removed the class allocators, though. You'll have to re-add these, if
needed. Please look at the file cpp_lex_token.hpp for reference).

> > >     * the line positioning is incorrect/non-standard (see 
> below for 
> > > an example).
> >
> > This does not depend on the Wave library itself, but it's a 
> matter of 
> > the real_positions example. Do you mind to submit a fix for that (I 
> > now changed the sample to output the detailed token 
> information, which 
> > makes it easier to track the output)?
> 
> Yes. It was very easy to work with your code so that should 
> be no problem. Are the real_positions set before or after the 
> tokens are send to the preprocessing hooks?

The corrected position is set inside the generated_token() pp hook. IIUC
correctly you're asking, whether the corrected token position will be
available in other hooks as well. The answer is no. The generated_token pp
hook is call at the very last point before the preprocessed token gets
returned to the calling application. I see no other way to do this, because
only at this point I know everything about the final token sequence. Any
ideas?

HTH
Regards Hartmut

> 
> Thanks
> Andrea
> 
> > > On 11/7/06, Hartmut Kaiser <hartmut.kaiser_at_[hidden]> wrote:
> > > >
> > > > Andreas,
> > > >
> > > > > I need some advice regarding Wave. What I want to do is
> > > to operate
> > > > > with Wave on a token stream and extract the token stream 
> > > > > representing macro calls after expansion. The twist is
> > > that I want
> > > > > the token stream after expansion to have the positions
> > > that another
> > > > > preprocessor would see on the already expanded output 
> from Wave.
> > > > > E.g:
> > > > > #define MACRO_CALL int x;
> > > > > int main(){
> > > > >      MACRO
> > > > > };
> > > > >
> > > > > Exands into
> > > > > #line 2
> > > > > int main(){
> > > > >      int x;
> > > > > };
> > > > >
> > > > > Where 'int x' should have a position of line 3 
> instead of line 1 
> > > > > after expansion. I also want the column number to
> > > represent what we
> > > > > see on line 3 instead of what we see on line 1. Is 
> this possible ?
> > > >
> > > > Ok, I've added a new pp hook (what else? :-P) to Wave:
> > > >
> > > >
> > > 
> ////////////////////////////////////////////////////////////////////
> > > //
> > > > /////
> > > > //
> > > > //  The function 'generated_token' will be called by 
> the library 
> > > > whenever a //  token is about to be returned from the library.
> > > > //
> > > > //  The parameter 'ctx' is a reference to the context
> > > object used for
> > > > //  instantiating the preprocessing iterators by the user.
> > > > //
> > > > //  The parameter 't' is the token about to be returned
> > > from the library.
> > > > //  This function may alter the token, but in this case it
> > > must be //
> > > > implemented with a corresponding signature:
> > > > //
> > > > //      Token const&
> > > > //      generated_token(Context const& ctx, Token& t);
> > > > //
> > > > //  which makes it possible to modify the token in place.
> > > > //
> > > > //  The default behavior is to return the token passed as the 
> > > > parameter //  without modification.
> > > > //
> > > >
> > > //////////////////////////////////////////////////////////////
> > > /////////////
> > > >     template <typename Context, typename Token>
> > > >     Token const&
> > > >     generated_token(Context const& ctx, Token const& t)
> > > >     { return t; }
> > > >
> > > > This should help solving your problem.
> > > >
> > > > To show how this hook function may help you I added a 
> new sample 
> > > > 'real_positions' to Wave demonstrating its use. Everything
> > > is in the
> > > > Boost CVS::HEAD.
> > > >
> > > > HTH
> > > > Regards Hartmut
> > > >
> > > > _______________________________________________
> > > > Boost-users mailing list
> > > > Boost-users_at_[hidden]
> > > > http://lists.boost.org/mailman/listinfo.cgi/boost-users
> > > >
> > > _______________________________________________
> > > Boost-users mailing list
> > > Boost-users_at_[hidden]
> > > http://lists.boost.org/mailman/listinfo.cgi/boost-users
> >
> > _______________________________________________
> > Boost-users mailing list
> > Boost-users_at_[hidden]
> > http://lists.boost.org/mailman/listinfo.cgi/boost-users
> >
> _______________________________________________
> Boost-users mailing list
> Boost-users_at_[hidden]
> http://lists.boost.org/mailman/listinfo.cgi/boost-users