Re: [boost] [wave] problem with long long suffixes

From: Hartmut Kaiser (hartmut.kaiser_at_[hidden])
Date: 2005-11-23 11:45:33

Paul Mensonides rote:

> > It would be ok if Wave tool printed this warning and preprocessing 
> > continued outputting two tokens '11' and 'LL'.
> > Instead of this it throws wave::cpplexer::lexing_exception 
> and stops 
> > preprocessing as if '11LL' is invalid character sequence.
> > '11LL' is in no way different from '11ABC' and the latter 
> produces two 
> > tokens as expected.
> 
> Both '11LL' and '11ABC' are valid _preprocessing tokens_.  
> However, with long long support, the former is a valid 
> _token_ but the latter is not.  At the point where 
> preprocessing tokens are converted into tokens, an error 
> should occur, not two tokens.  That point is directly between 
> where preprocessing ends and the rest of the phases of 
> translation.  Normally, a standalone preprocessor doesn't go 
> all the way in the conversion phase, but Wave does because it 
> is producing tokens by going into phase 7.  IMO, there are 
> only two options here: 1) Wave should yield preprocessing 
> tokens instead of tokens, or 2) Wave should yield an error 
> trying to translate '11ABC' into a token (or '11LL' if long 
> long is not explicitly enabled--it isn't part of standard C++ yet).

Since Wave produces tokens and not preprocessing tokens I believe the
current behaviour is the correct one.

> > This kind of behavior leads to some problems. For example, 
> you can't 
> > use Boost.Wave for preprocessing many of the boost files without 
> > enabling support for long long suffixes because 
> boost/limit.hpp uses 
> > them (but they are ifdef'ed away if you define some macroses).
> 
> Personally, I think that Wave should produce preprocessing 
> tokens.  An iterator adaptor could be applied to produce 
> tokens (or more accurately, do the checking that tokens require).

Will have to think about this. One of the goals of the Wave library was not
only to make available a preprocessor but also lexer components producing
C++ tokens usable in other contexts as well. 

Wave currently doesn't follow strictly the mandated compilation phases. This
has to be changed in the future anyway.

Regards Hartmut