Boost logo

Boost Users :

From: Tom Browder (tbrowder_at_[hidden])
Date: 2005-06-11 12:52:56

I have used my own C++ tokenizer in the past, but I would like to use
Boost's instead.

The predominant use of tokenizing for me is to split on white space, but
Boost's default is to use white space AND punctuation. Is there any
possibility to have either the default changed, or another TokenizerFunction
added such as ws_separator, or something similar?

I know I can use

  boost::char_separator<char> sep(" \n\t");

(but do I need to add "\v" to the char set?)

but I would rather have something like

  boost::ws_separator sep;

and, better, make the ws_separator be the default TokenizerFunction for

Thanks for listening.

Tom Browder

Boost-users list run by williamkempf at, kalb at, bjorn.karlsson at, gregod at, wekempf at