Re: Third update to Token Iterator concepts and code

From: jbandela_at_[hidden]
Date: 2000-09-13 12:04:14

> It is referenced in the main tokenizer.hpp file since (a version of)
> punct_space_tokenizer is used as the default type in the second 
template
> argument of the token_iterator class.  I think token_iterator 
should be just
> another sample, and not the default tokenizer.  It's highly likely 
that some
> other tokenizer will be used, so not including 
punct_space_tokenizer and its
> owning header file means clients who don't use it won't pay for 
having it.

I agree with you there. What you do not use, you do not pay for is an 
underlying principle of C++. However, I believe we can leave the 
default TokenizerFunc = punct_space_tokenizer<Iter> argument in 
token_iterator, and remove the #include statement. If they want to 
use the default, they can just include the punct_space_tokenizer 
file. Otherwise, they can just leave it out and specify another 
tokenizer.

> Maybe some sort of traits class should be added.  The 
ptr_tokenizer_fun
> would use that traits class, and so can any other potential 
tokenizer or
> adapter.

I am not sure, I exactly understand what you saying. Would you please 
provide some sample code that illustrates this.

> It choked on me, see below.

Go ahead and add a default, do-nothing constructor to the tokenizer. 
It probably won't hurt anything. One thing about this tokenizer, is 
that it is just a sample, and does not correctly handle the boundary 
conditions where either what,with or both are empty strings. As this 
is just a demo and not production code, I have not felt it was too 
bad. However, I have created an updated version that fixes this 
problem. I may post it later, if enough people ask me to.

Thanks for your input, and let me know if you can compile it with 
replace_tokenizer

John R. Bandela