Tokenizer

From: macrodisk (macrodisk_at_[hidden])
Date: 2002-04-03 20:17:49

I have a question about boost::tokenizer. I have a Semicolon separated
string, something like:
"All;;;;;;Bus;;;;;;BRT;;;;;;All"
When I use boost tokenizer, I get "All", "Bus", "BRT",and "All". I 
want to get the the empty string between the semicolons, seems like 
the tokenizer takes consecutive semicolons as a single token; is 
there any way I can change this to represent different strings?

boost::char_separator<char> sep(";");
  typedef boost::tokenizer<boost::char_separator<char> >
      tokenizer;
tokenizer Id_tokens(IdString, sep);
  for(tokenizer::iterator Id_tok_iter= Id_tokens.begin(), 
End_Id_tok_iter =
Id_tokens.end();
        Id_tok_Iter != End_Id_tok_iter)
{
    //Do Something
}

Another question, is there some way to assign a different string once 
a tokenizer is constructed?

tokenizer Id_tokens(IdString,sep);
Id_tokens.assign(NewString,sep); //Is this possible with the boost
tokenizer?

I couldn't find any examples to this end in the places I searched.

Thank you,

Vijay