Re: [Boost-users] Tokenizer Question

From: Victor A. Wagner Jr. (vawjr_at_[hidden])
Date: 2005-06-11 18:14:08

At 10:52 2005-06-11, Tom Browder wrote:
>I have used my own C++ tokenizer in the past, but I would like to use
>Boost's instead.
>
>The predominant use of tokenizing for me is to split on white space,

for tokenizing on whitespace, simple stream input (>>) to a std::string 
suffices.
IMO, it's hardly worth troubling yourself with a tokenizer for whitespace.

>  but
>Boost's default is to use white space AND punctuation.  Is there any
>possibility to have either the default changed, or another TokenizerFunction
>added such as ws_separator, or something similar?
>
>I know I can use
>
>   boost::char_separator<char> sep(" \n\t");
>
>(but do I need to add "\v" to the char set?)
>
>but I would rather have something like
>
>   boost::ws_separator sep;
>
>and, better, make the ws_separator be the default TokenizerFunction for
>tokenizer.
>
>Thanks for listening.
>
>Tom Browder
>
>
>
>
>
>
>_______________________________________________
>Boost-users mailing list
>Boost-users_at_[hidden]
>http://lists.boost.org/mailman/listinfo.cgi/boost-users

Victor A. Wagner Jr.      http://rudbek.com
The five most dangerous words in the English language:
               "There oughta be a law"