Re: [Boost-Users] Tokenizer

From: Jeremy Siek (jsiek_at_[hidden])
Date: 2002-04-03 23:09:04

In the constructor of the char_separator, you need to specify
keep_empty_tokens. See

http://www.boost.org/libs/tokenizer/char_separator.htm

On Thu, 4 Apr 2002, macrodisk wrote:

macrod> I have a question about boost::tokenizer. I have a Semicolon separated
macrod> string, something like:
macrod> "All;;;;;;Bus;;;;;;BRT;;;;;;All"
macrod> When I use boost tokenizer, I get "All", "Bus", "BRT",and "All". I
macrod> want to get the the empty string between the semicolons, seems like
macrod> the tokenizer takes consecutive semicolons as a single token; is
macrod> there any way I can change this to represent different strings?
macrod>
macrod> boost::char_separator<char> sep(";");
macrod>   typedef boost::tokenizer<boost::char_separator<char> >
macrod>       tokenizer;
macrod> tokenizer Id_tokens(IdString, sep);
macrod>   for(tokenizer::iterator Id_tok_iter= Id_tokens.begin(),
macrod> End_Id_tok_iter =
macrod> Id_tokens.end();
macrod>         Id_tok_Iter != End_Id_tok_iter)
macrod> {
macrod>     //Do Something
macrod> }
macrod>
macrod> Another question, is there some way to assign a different string once
macrod> a tokenizer is constructed?
macrod>
macrod> tokenizer Id_tokens(IdString,sep);
macrod> Id_tokens.assign(NewString,sep); //Is this possible with the boost
macrod> tokenizer?
macrod>
macrod> I couldn't find any examples to this end in the places I searched.
macrod>
macrod> Thank you,
macrod>
macrod> Vijay
macrod>
macrod>
macrod>
macrod>
macrod>
macrod> Info: <http://www.boost.org>
macrod> Wiki: <http://www.crystalclearsoftware.com/cgi-bin/boost_wiki/wiki.pl>
macrod> Unsubscribe: <mailto:boost-users-unsubscribe_at_[hidden]>
macrod>
macrod>
macrod> Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/
macrod>
macrod>

----------------------------------------------------------------------
 Jeremy Siek                          http://php.indiana.edu/~jsiek/
 Ph.D. Student, Indiana Univ. B'ton   email: jsiek_at_[hidden]
 C++ Booster (http://www.boost.org)   office phone: [phone]
----------------------------------------------------------------------