Re: Review: Tokenizer

From: jbandela_at_[hidden]
Date: 2001-06-03 19:57:15

> 
>   token_iterator.hpp
>     token_iterator_generator
>     make_token_iterator
>       (I did not list token_iterator here because that class should 
be
>        deleted, see below)
> 
>   tokenizer.hpp
>     tokenizer
> 
>   token_functions.hpp
>     cvs_separator
>       csv_escape_error  (there's a typo in the name of this one)
>     offset_separator
>     punct_space_separator
> 
> Implementation details (namespace detail)
> 
>   token_iterator.hpp
>     tokenizer_policy
>     tokenizer_base  (should be renamed token_iterator_base)
> 
> 

So you are basically saying to rename tokenizer_policy.hpp to 
tokenizer_iterator.hpp and put tokenizer_policy in detail?

> The token_iterator class should be deleted. You are attempting to 
create a
> templated typedef using inheritance, which I'm afraid doesn't 
really work.
> There are corner cases where bad things happen. Also, the
> token_iterator_generator already serves the purpose.

Could you give some examples of when this would cause problems? Right 
now I see it as a rather nice convenience class, but I am willing to 
be enlightened.

> There are a couple additions I'd like to see to the library. We 
need a
> tokenizer function that provides the same kind of functionality as
> strtok(). I wrote up a version of this a while back. I'll dust that 
code
> off, write docs, and then post it. Also I'd like to see a tokenizer
> function that uses a regular expression (from regex) to do the 
spliting.
> 

Doesn't punct_space_separator provide all the functionality in 
strtok, namely using single characters as delimiters? However, I 
think having the extended functionality of your class would be a 
bonus. As to the regex tokenizer function, I do not have much 
experience with regex and regular expressions, so I probably would 
not be the one to write it. If someone could write one, I would be 
glad to include it in the functions.

> On the positive side. I must say I really like the idea of 
parameterizing
> the token iterator with the tokenizing function. This design is 
really
> nice.

Thanks for your comments,
John R. Bandela