Re: [boost] Re: Review: Tokenizer

From: Jeremy Siek (jsiek_at_[hidden])
Date: 2001-06-04 14:06:18

On Mon, 4 Jun 2001 jbandela_at_[hidden] wrote:
>
> Actually, you can pass in the characters that should be considered
> punctuation and whitespace into the constructor( they are just
> defaulted). The difference is that punctuation can be returned as a
> token, while whitespace cannot be returned as a token.

Ahh, I missed that constructor. Perhaps the name of punct_space_separator
could be improved... because of the name I didn't expect to find the more
general functionality. Perhaps something like char_separators would be
better.

Cheers,
Jeremy

----------------------------------------------------------------------
 Jeremy Siek                        www: http://www.lsc.nd.edu/~jsiek/
 Ph.D. Candidate, IU B'ton          email: jsiek_at_[hidden]
 Summer Manager, AT&T Research      phone: [phone]
----------------------------------------------------------------------