Layered Iterators and orthogonality (was Layered Iterator Concept)

From: jbandela_at_[hidden]
Date: 2000-09-14 10:25:37

I liked the layered iterator concept. However, other than a name 
change, I do not see what it really adds to token_iterator. As I 
looked at token_iterator, I realized it is a bad name. It is actually 
a bad name. token_iterator and concept, is actually a generalized 
concept of a layered forward iterator that depends on a transforming 
function. 

The VTL is neat, but I believe it suffers from coupling the 
view/container/iterator concept, too close to the algorithm that 
produces them. Thus you have a different view for each manipulating 
algorithm.

What I propose is a family of layered iterators based on various 
(input/forward, bidirectional, and random).

The Forward Layer Iterator is are refinement of input/forward 
depending on what the underlying iterator is.

It depends on 

bool TransformFunc::operator()(iterator& current, iterator end, 
TransformedType& out)

which is basically the Tokenizer Concept

Next we have Bidirectional Layered Iterator which is a refinement of 
both Bidirectional Iterator and Forward Layered Iterator.

For ++ it uses the same function as Forward Layer Iterator uses
For -- it uses
bool TransformFunc::operator()(iterator& current, iterator end, 
iterator begin, TransformedType& out)

This has roughly the same semantics as TokenizerFunc but finds the 
previous transformed "token"

Finally we have Random Layered Iterator which is a refinement of 
Bidirectional Layered Iterator
For + and += and - and -= it uses

bool TransformFunc::operator()(iterator& current, iterator end, 
iterator begin, int n, TransformedType& out)

which has roughly the same semantics as the previous functor, except 
that n specifies the distance and the sign of n specifies the 
direction.

Now we have several concepts related to TransformFunc
They are grouped by Layered Iterator support, and underlying iterator 
support.

They are

I. Forward, X
These all implement

bool TransformFunc::operator()(iterator& current, iterator end, 
iterator begin, TransformedType& out)

X is the underlying iterator. THe most restrictive is input, the most 
permissive is random. A TransformFunc that can work with input 
underlying iterator, can also work with forward,bidirectional, and 
random.

II. Bidirectional,X
Refinement of Forward,X
also implements

bool TransformFunc::operator()(iterator& current, iterator end, 
iterator begin, TransformedType& out)

It can work with both Bidirectional Layered Iterator and Forward 
Layered Iterator, the underlying iterator must at least be of  type 
X, and it will work with more refined iterators than type X (if X is 
input, it will also work with forward,bidi,random)

III. Random, X
Refinement of Forward,X
also implements

bool TransformFunc::operator()(iterator& current, iterator end, 
iterator begin, int n, TransformedType& out)

It can work with Random Layered Iterator,Bidirectional Layered 
Iterator, and Forward Layered Iterator. The underlying iterator must 
at least be of  type X, and it will work with more refined iterators 
than type X (if X is input, it will also work with 
forward,bidi,random)

----
Thus we have orthagonality between Layered Iterators and the 
TransformFunc's they depend on.
A Random,Input TransformFunc can be used with every Layered Iterator
Examples
token_iterator is an example of Forward Layered Iterator
punct_space_tokenizer is an example of Forward,Input TransformFunc
By using this method, we decouple the iterator functionality from the 
filtering/transforming functionality and thus can achieve greater 
reuse.
Sorry about the long post, but tell me what you think
John R. Bandela