Re: Review: Tokenizer

From: jbandela_at_[hidden]
Date: 2001-06-04 13:50:18

> One of the main problems is that the return type of operator++ is
> wrong, it isn't token_iterator. Here's an example:

I see the problem now. Thanks.

> > Doesn't punct_space_separator provide all the functionality in
> > strtok, namely using single characters as delimiters? However, I
> 
> I don't think so. strtok() lets the user specify the delimiters, 
whereas I
> thought pucht_space_separator hard-codes in the separators as 
punctuation
> and whitespace.
> 

Actually, you can pass in the characters that should be considered 
punctuation and whitespace into the constructor( they are just 
defaulted). The difference is that punctuation can be returned as a 
token, while whitespace cannot be returned as a token.

Cheers,
John R. Bandela