Re: [Boost-users] String Tokenizer question

From: Bjorn.Karlsson_at_[hidden]
Date: 2005-07-12 05:29:41

> But what I want to get is "my", "birthday is 1976", "15" and "17".
> In other words, I want the tokenizer either can give us the remaining of
the 
> string or can change the delimiters dynamically.
> Is there a way to do this?

You need to use two tokenizers, just like in the example you provided. The
only thing you need to do differently is retrieve an iterator to the current
position in the input string after the first tokenizer iterator (beg) has
been dereferenced, and pass the iterator range [current,end) to the
constructor of the tokenizer next rather than the whole string (s). You can
retrieve the current iterator from a token_iterator by calling the
iterator's member function base(). Here's your example again, modified to do
just that:

  std::string s = "my birthday is 1976, 15/17";
  boost::tokenizer<> tok(s);
  boost::tokenizer<>::iterator beg=tok.begin();
  // here *beg will give us "my"

  typedef boost::tokenizer<boost::char_separator<char> >
    Boost_char_tokenizer;

  boost::char_separator<char> sep(",/");
  Boost_char_tokenizer next(beg.base(),s.end(), sep);
  Boost_char_tokenizer::iterator tok_iter = next.begin();
  for(; tok_iter!=next.end(); ++tok_iter){
    std::cout << *tok_iter << std::endl;
  }
  // this will give us " birthday is 1976", " 15" and "17"

Note that the only difference in this version is the construction of next:

  Boost_char_tokenizer next(beg.base(),s.end(), sep);

Bjorn