[Boost-users] Tokenizer Question

From: Tom Browder (tbrowder_at_[hidden])
Date: 2005-06-11 12:52:56

I have used my own C++ tokenizer in the past, but I would like to use
Boost's instead.

The predominant use of tokenizing for me is to split on white space, but
Boost's default is to use white space AND punctuation.  Is there any
possibility to have either the default changed, or another TokenizerFunction
added such as ws_separator, or something similar?

I know I can use

  boost::char_separator<char> sep(" \n\t");

(but do I need to add "\v" to the char set?)

but I would rather have something like

  boost::ws_separator sep;

and, better, make the ws_separator be the default TokenizerFunction for
tokenizer.

Thanks for listening.

Tom Browder