[boost] Tokenizer Improvements

From: UF (jbandela_at_[hidden])
Date: 2002-07-04 13:23:39

I have recently made some changes to tokenizer. Among the changes are using
tok.assign in some TokenizerFunctions and sharing 1 TokenizerFunction in a
tokenizer with all its iterators. Please see
http://plaza.ufl.edu/jbandela/tokenizer/ for more information as well as to
download and try the new changes. I would like some feedback on these
changes before putting them into the CVS.

Regards,

John R. Bandela