[Boost-users] Tokenizer - remainder of parsed string

From: Gleske, Paul GOSS (Paul.Gleske_at_[hidden])
Date: 2007-04-02 18:23:54

I have a situation where I want to parse off the first few fields of a
message and then return the rest of the message.   The message format
is:
 
<STX>field1,field2,field3,field4,field5,field6,field7,field8,field9<ETX>
 
What I want to do is pull off the first 5 fields and then get at the
remainder of the string (fields 6-9) as a single string.
 
I've pieced together something that mostly works  It leaves me with a
remainder of ",field6,field7,field8,field9".  Note the leading comma.
 
-----------------
 std::string str = "\x02"
"field1,field2,field3,field4,field5,field6,field7,field8,field9" "\x03";

 typedef boost::tokenizer<boost::char_separator<char> > tokenizer;
 boost::char_separator<char> sep(",\x02\x03");
 tokenizer tokens(str, sep);
 tokenizer::iterator tok_iter = tokens.begin();
 for (int i=1; tok_iter != tokens.end(); tok_iter++)
 {
  std::cout << "<" << *tok_iter << "> ";
  ++i;
  if (i > 5)
   break;
 }
 std::cout << "\n";
 
 std::cout << "current token <" << tok_iter.current_token() << ">";
 std::cout << "\n";
 
 boost::char_separator<char> sep2("\x03");
 tokenizer remainder(tok_iter.base(), str.end(), sep2);
 
 std::cout << "<" << *remainder.begin() << ">" << "\n";

-----------------
 
What I don't understand is what the tok_iter.base() returns, or how the
tokenizer constructor that uses it works (see the line for remainder).
How can that constructer take tok_iter.base() and str.end()?  Is the
string iterator converted?  
 
Is there a better way to get the remainder of the tokenized string?  I
would very much like to not have the leading comma on the remainder.  I
know I can use substr on the remainder, but it seems that there should
be a better way.
 
Thanks,
 
-pkg
 
--------------------------------------------------------

Goss ... Innovation for Business

NOTICE: This e-mail and any attachment(s) may contain confidential and proprietary information of Goss International Corporation and/or its subsidiaries and may be legally privileged. This e-mail is intended solely for the addressee. If you are not the addressee, dissemination, copying or other use of this e-mail or any of its content is strictly prohibited and may be unlawful. If you are not the intended recipient please inform the sender immediately and destroy the e-mail and any copies. All liability for viruses is excluded to the fullest extent permitted by law. Any views expressed in this message are those of the individual sender. No contract may be construed by this e-mail.