Tokenizer Policy Class

From: jbandela_at_[hidden]
Date: 2001-04-10 19:35:37

I took the old token_iterator class and re-wrote it as a policy class 
for use with iterator_adapter. There is also a generator template and 
a creator function, as with the other policy classes. The policy 
class uses the same TokenizerFuncs as before. In addition, as a 
convenience, there is a token_iterator class that is implemented as a 
8 or so line derivation of iterator_adapter.

Take a look at it and tell me what you think.  It is in the directory 
Tokenizer Policy.