[Boost-users] Tokenizer Problem

From: Thomas Johnson (thomas_at_[hidden])
Date: 2004-02-21 06:31:49

Can anyone explain to me why the code below prints:
Token: A1B21
Token: A1B
Token: :
Token: 12

Instead of printing:
Token: A1B2
Token: :
Token: 12
Token: A1B
Token: :
Token: 12

Using boost 1.31.0
Thanks!

---Cut here---
#include <boost/tokenizer.hpp>
#include <iostream>
                                                                                                                                  
int main() {
  boost::char_separator<char> sep("", ":", boost::keep_empty_tokens);
  boost::tokenizer< boost::char_separator<char> > stringTok(std::string("A1B2:12"),sep);
  boost::tokenizer< boost::char_separator<char> >::iterator tokIter=stringTok.begin();
                                                                                                                                  
  while(tokIter!=stringTok.end()) {
    std::cout << "Token: " << (*tokIter) << "\n";
    tokIter++;
  }
                                                                                                                                  
  boost::tokenizer< boost::char_separator<char> > stringTok2(std::string("A1B:12"),sep);
  boost::tokenizer< boost::char_separator<char> >::iterator tokIter2=stringTok2.begin();
                                                                                                                                  
  while(tokIter2!=stringTok2.end()) {
    std::cout << "Token: " << (*tokIter2) << "\n";
    tokIter2++;
  }
}
---Cut Here---

-- 
Thomas Johnson <thomas_at_[hidden]>