Re: [Boost-users] [WAVE] position of tokens

From: Hartmut Kaiser (hartmut.kaiser_at_[hidden])
Date: 2006-11-09 18:37:17

Andreas, 

> > I changed the real_positions example to use a new token 
> type carrying 
> > both, the original and the corrected positions. This makes it a bit 
> > more tricky, though, because now you have to explicitely 
> instantiate 
> > some of the internal Wave library template types for this new token 
> > type. These explicit instantiations are normally done in the Wave 
> > library code (for the default token type).
> 
> I think this probably is the optimal solution and it is not 
> difficult to instantiate the templates in my library although 
> I expect it to slow down compilation considerably (?).

These need to be recompiled only if you have changed the token class, which
should happen rarely.

> > Note: the new token type is essentially a copy of the default token 
> > type which has added the second position instance. For simplicity 
> > reasons I removed the class allocators, though. You'll have 
> to re-add 
> > these, if needed. Please look at the file cpp_lex_token.hpp 
> for reference).
> 
> In which instances would I need to use these class 
> allocators? What are they used for in Wave with the standard 
> token type?

Yes, the default token type in Wave uses a boost::pool based class allocator
to improve performance. Please compare the real_positions_token and the
default Wave token type in the file cpp_lex_token.hpp.

> > > Yes. It was very easy to work with your code so that should be no 
> > > problem. Are the real_positions set before or after the 
> tokens are 
> > > send to the preprocessing hooks?
> >
> > The corrected position is set inside the generated_token() pp hook. 
> > IIUC correctly you're asking, whether the corrected token position 
> > will be available in other hooks as well. The answer is no. The 
> > generated_token pp hook is call at the very last point before the 
> > preprocessed token gets returned to the calling 
> application. I see no 
> > other way to do this, because only at this point I know everything 
> > about the final token sequence. Any ideas?
> 
> An idea would be to allow calling of generated_token() from 
> one of the other hooks. For instance, if you in 
> rescanned_macro() know when the first macro has been fully 
> texpanded then you can call
> generated_token() at that point on every token provided to 
> rescanned_macro(). The reason why I say first macro is that a 
> macro in it's formal definition can call another macro, so 
> first macro calls second macro etc.
> 
> The suggestion I made above can probably also be considered 
> an optimization (?) because you could also imagine 
> recomputing the correct positions on every call to rescanned_macro.
> Do you see any reason for this not working?

If the generated_token() gets call for instance in the rescanned_macro()
hook you won't be able to track the resulting position of the token because
it's impossible to tell, at which expansion level inside a macro the current
token got generated. Do I miss something?

Regards Hartmut