Re: Tokenizer review comments

From: jbandela_at_[hidden]
Date: 2001-06-05 14:47:05

--- In boost_at_y..., Gary Powell <Gary.Powell_at_s...> wrote:
> I was looking over my IOStreams book on the use of narrow and 
widen, and it
> occurred to me that tokenizer should also be using the locale stuff.
> 
> Now take all of this with a large grain of salt, as by no means am 
I an
> expert in the local/facet area of C++.
> 
> But as I read this stuff I was wondering if you could use the is
(punct,c) as
> the default instead of specifying the characters yourself. As in 
testing
> ctype<Char>::is(ctype_base::punct, c); ? (same goes for 
whitespace.) (I
> think its ctype_base::space )
> 

From my documentation. "is" is non-static. I like your idea as it 
could get rid of the punct_and_space template. For example with 
punctuation, we could do this.

In the constructor
punct_space_tokenizer(...,char* punct = "", char* space = "")

Then in IsPunct
std::basic_string<Char>::iterator beg(punctuation_.begin()),
end(punctuation_.end());
if(beg==end){
ctype<Char> ct;
return ct.is(ctype_base::punct,c);
}
else{
return find(beg,end,c);
}