Re: [Boost-users] [WAVE] position of tokens

From: Hartmut Kaiser (hartmut.kaiser_at_[hidden])
Date: 2006-11-10 08:50:48

Andreas, 

> > If the generated_token() gets call for instance in the 
> > rescanned_macro() hook you won't be able to track the resulting 
> > position of the token because it's impossible to tell, at which 
> > expansion level inside a macro the current token got 
> generated. Do I miss something?
> 
> You know that for each call to the hook expand_*_like_macro() 
> there is one call to rescanned_macro(), so by using this 
> information I think that you can know the epansion level. In 
> order to avoid multiple calls to generated_token() it is 
> possible to set a bit in the token which tells that it was 
> already set or maybe the correct positions should be negative 
> to begin with(?). Do you see any problems with such an approach?

Even if macro expansion in Wave is implemented as a purely recursive
process, it isn't one in reality. Is more like a gliding window moving over
the input token stream steadily producing tokens. I see no way to determine
(at least not yet - will have to think about it), whether a certain token
'is finished' or if it still will get modified by a pending rescanning step.
In any case the correct position information will be available only _after_
the last rescanning.

The question is, when exactly you _need_ the corrected position information
of the tokens. If you need to access the corrected token information only
_after_ the preprocessing, my current approach will work fine, because even
if you store some of the tokens during macro expansion and the position
information gets corrected only later, the corrected position information
will be available. The reason is, that the real_positions_token stores the
position information 'by reference' which makes it available to stored
copies of the same token. 

Does this make sense?

Regards Hartmut