Third update to Token Iterator concepts and code

From: jbandela_at_[hidden]
Date: 2000-09-08 22:22:01

I have updated the Token Iterator and Tokenizer concepts. It is 
located in BoostUpdate3.zip in TokenIterator directory

In addition, I have provided an implementation of those concepts 
along with several samples. The code has been reorganized. Here are 
some highlights of the update.

1. punct_space_tokenizer moved to separate file since there are many 
ways to implement it.

2. New template ptr_tokenizer_fun. This is analogous to ptr_fun in 
the STL and is used to turn a regular function into a Tokenizer

3. Some of the sample tokenizers that don't need to maintain state 
informationn are now functions and use ptr_tokenizer_fun

4. New samples
a) A simple tokenizers that filters out non-alpha characters
b) A simple 4 function (no operator precedence) calculator
c) A replacing tokenizer

Of these, I believe c) is the most interesting. You give it a string 
to replace and what to replace it with, and when it is incremented 
and dereferenced, it will return the character that would have been 
there had a search/replace been done. However, it does it without 
modifying the orignal sequence. It is also capable of being used with 
input iterators. To do such, because it needs to scan ahead, it has 
to maintain a buffer. However, the max size of the buffer is the size 
of the string to replace with.  Check it out and tell me what you 
think.

Thanks to everyone who has taken the time to look at the code and 
concepts and provide feedback. Please let me know what you think of 
this update including concepts, code, and samples.

John R. Bandela