Re: Tokenizer formal review

From: Rémi Delcos (remi.delcos_at_[hidden])
Date: 2001-06-04 02:55:43

Hello John,

Two small issues when compiling with MWCW Pro 6.2 on Mac OS:

- tokenizer_policy.hpp line 77: Qualify ptrdiff_t with std::

- token_functions.hpp line 249-250: The ';' after 'if(...)' generates 
a "possibly unwanted ';'" warning. Replacing ';' with '{}' solves the 
problem.

A few other comments:

Maybe <cassert> could be replaced by something more flexible because 
its default behavior doesn't make sense in all applications. For 
example I use my own asserts which will drop into a symbolic debugger.

When using csv_separator, I would like to get an empty token if the 
input ends with a token separator, i.e. ",one,two," -> {empty, 'one', 
'two', empty}.

I feel that csv_separator and whitespace_and_punct while being useful 
are maybe not general enough to be part of a library. Especially 
whitespace_and_punct which has white space characters and the two 
kinds of punctuations hard coded. It would be nice to have the 
separator characters parametrizable. Some sensible default values 
would permit out of the box use.

Even with these issues, I would like to see the tokenizer library 
added to boost.

Rémi