Re: Tokenizer review comments

From: jbandela_at_[hidden]
Date: 2001-06-01 18:23:48

> Code comments / questions:
> 1) I would have expected that the Tokenizer class would have 
> 
>   template <class TokenizerFunc,
>   		class Token=std::string,
> 	      class Iterator=std::string::const_iterator>
>   class Tokenizer {...}
> 
> 
> Then the third example could be simplified from:
>   typedef 
tokenizer<string,string::const_iterator,punct_space_separator<char> >
> Tok;
> to
>   typedef tokenizer<punct_space_separator<char> > Tok;
> 
> In my mind this is much clearer.

I like your idea, as it will simplify the common case where a string 
is being is being broken up. 

> 2) Is it possible to use a raw cstring instead of std::string as 
the input?  I
> experimented a bit, but I was unsuccessful in getting this to 
work.  I have a
> use for this in an application which needs to tokenize data from a 
socket
> connection which is returned as a cstring and I wouldn't want the 
overhead of an
> extra string construction / string copy.
> 

Yes. Use const char* instead of string::const_iterator as the 
iterator argument. Leave all the other parameters the same. It will 
take the cstring and break it up into a series of strings.

Thanks for your comments,

John R. Bandela