Re: [Boost-users] String Tokenizer question

From: J Liu (johnny_spring_at_[hidden])
Date: 2005-07-12 09:43:27

This works.
Thank you very much!
<Bjorn.Karlsson_at_[hidden]> wrote in message 
news:3D8559AE95B4D611B02C0002557C6C8B01A1AB76_at_STH-EXCH...
>
>> But what I want to get is "my", "birthday is 1976", "15" and "17".
>> In other words, I want the tokenizer either can give us the remaining of
> the
>> string or can change the delimiters dynamically.
>> Is there a way to do this?
>
> You need to use two tokenizers, just like in the example you provided. The
> only thing you need to do differently is retrieve an iterator to the 
> current
> position in the input string after the first tokenizer iterator (beg) has
> been dereferenced, and pass the iterator range [current,end) to the
> constructor of the tokenizer next rather than the whole string (s). You 
> can
> retrieve the current iterator from a token_iterator by calling the
> iterator's member function base(). Here's your example again, modified to 
> do
> just that:
>
>  std::string s = "my birthday is 1976, 15/17";
>  boost::tokenizer<> tok(s);
>  boost::tokenizer<>::iterator beg=tok.begin();
>  // here *beg will give us "my"
>
>  typedef boost::tokenizer<boost::char_separator<char> >
>    Boost_char_tokenizer;
>
>  boost::char_separator<char> sep(",/");
>  Boost_char_tokenizer next(beg.base(),s.end(), sep);
>  Boost_char_tokenizer::iterator tok_iter = next.begin();
>  for(; tok_iter!=next.end(); ++tok_iter){
>    std::cout << *tok_iter << std::endl;
>  }
>  // this will give us " birthday is 1976", " 15" and "17"
>
>
> Note that the only difference in this version is the construction of next:
>
>  Boost_char_tokenizer next(beg.base(),s.end(), sep);
>
>
> Bjorn