Re: Tokenizer formal review

From: jbandela_at_[hidden]
Date: 2001-06-04 13:52:53

> are maybe not general enough to be part of a library. Especially 
> whitespace_and_punct which has white space characters and the two 
> kinds of punctuations hard coded. It would be nice to have the 
> separator characters parametrizable. Some sensible default values 
> would permit out of the box use.

White space and punctuation characters can be passed into the 
punct_space_separator constructor. The template specialization serves 
only to provide the defaults for these parameters.

Cheers,
John R. Bandela