Re: Concepts for Token Iterator and Tokenizer

From: jbandela_at_[hidden]
Date: 2000-09-01 00:40:59

I see only three functions that are not related to iterators. Other 
than that, the functions are either related to stuff necessary for 
C++ classes (construction, assignment, etc) or iterators 
(dereferencing, comparison, etc). Of the three functions 
(next,end,valid), next and end are sometimes useful when you only 
want to parse a predefined number of tokens and then get the rest of 
the sequence (for example ini files where you have entries like
public key=adeechdh=223adkjf
you want grab the the public key and the = but grab the rest as a 
whole without parsing). As for the valid function, there is really no 
case where you really need it except as a debugging aid. The main 
reason for it, is made the semantics of some operations easier to 
describe (for example == ). In addition, we need to have a valid 
flag, since we cannot count on the value of an iterator we did not 
initialize( for example when we construct the end token_iterator a la 
istream_iterator).

To separate next end, implementing them in another class would be 
difficult. valid however would be easy since you could write it 
something like this
template<class T>
valid(const T& token_it){
return token_it != T();
}

I hope this rationale helps. To summarise, one of the main reasons is 
that having these functions makes it easier to document the semantics 
and post-conditions of several iterator related operations.

Thanks for your comments

John Bandela

--- In boost_at_[hidden], jsiek_at_l... wrote:
> 
> This will be a timely addition... one of the GGCL examples does some
> tokenizing.
> 
> One complaint about the design. The TokenIterator has lots of
> non-iterator operations. This degrades the conceptual integrity of 
the
> design.  Would it be possible to seprate out the functionality into
> more objects, where one object is *just* and iterator, and the rest 
of
> the functionality is in another object?
> 
>