Boost logo

Boost :

From: Rémi Delcos (remi.delcos_at_[hidden])
Date: 2001-06-04 02:55:43

Hello John,

Two small issues when compiling with MWCW Pro 6.2 on Mac OS:

- tokenizer_policy.hpp line 77: Qualify ptrdiff_t with std::

- token_functions.hpp line 249-250: The ';' after 'if(...)' generates
a "possibly unwanted ';'" warning. Replacing ';' with '{}' solves the

A few other comments:

Maybe <cassert> could be replaced by something more flexible because
its default behavior doesn't make sense in all applications. For
example I use my own asserts which will drop into a symbolic debugger.

When using csv_separator, I would like to get an empty token if the
input ends with a token separator, i.e. ",one,two," -> {empty, 'one',
'two', empty}.

I feel that csv_separator and whitespace_and_punct while being useful
are maybe not general enough to be part of a library. Especially
whitespace_and_punct which has white space characters and the two
kinds of punctuations hard coded. It would be nice to have the
separator characters parametrizable. Some sensible default values
would permit out of the box use.

Even with these issues, I would like to see the tokenizer library
added to boost.


Boost list run by bdawes at, gregod at, cpdaniel at, john at