Update to Token Iterator and Tokenizer concepts

From: jbandela_at_[hidden]
Date: 2000-09-05 21:01:34

I have updated these concepts. The main change is that the Tokenizer 
operator() returns an iterator to the beginning of where in the 
sequence the current token corresponds to the sequence. The previous 
concept, returned the past the end of the last element in the 
sequence that corresponded to the current token. In addition, next is 
now a reference that is updated by the functor.

The problem with the old approach was that it could not detect when 
the sequence had no tokens, but was non-empty.

Finally, thanks to all of you who have commented. I am seeing there 
seems to be some concern about complexity. In the current 
implemention (not posted yet, will be posted after the concepts are 
nailed down) I have moved all the tokenizers to a different file from 
the core token_iterator. The core token_iterator class is now 101 
lines including whitespace and comments. In addition, a new template 
ptr_tokenizer_fun<Iter,TokType> has been added. It is analogous to 
the standard ptr_fun. It allows an ordinary function that has the 
right parameters and return value to be turned into a Tokenizer. In 
addition, new sample Tokenizers have been implemented. These include 
one that only filters out non-alpha characters. Another is a 
calculator parsing tokenizer. This is used to implement a simple four 
function (non-operator precedence) calculator. In addition, these 
tokenizers are both implemented as functions and use the 
ptr_tokenizer_fun.

Thanks again for your time.

John Bandela