[Boost-users] Tokenizer Problem

21 Feb 2004

      Can anyone explain to me why the code below prints:
Token: A1B21
Token: A1B
Token: :
Token: 12

Instead of printing:
Token: A1B2
Token: :
Token: 12
Token: A1B
Token: :
Token: 12

Using boost 1.31.0
Thanks!

---Cut here---
#include <boost/tokenizer.hpp>
#include <iostream>

int main() {
  boost::char_separator<char> sep("", ":", boost::keep_empty_tokens);
  boost::tokenizer< boost::char_separator<char> > stringTok(std::string("A1B2:12"),sep);
  boost::tokenizer< boost::char_separator<char> >::iterator tokIter=stringTok.begin();

  while(tokIter!=stringTok.end()) {
    std::cout << "Token: " << (*tokIter) << "\n";
    tokIter++;
  }

  boost::tokenizer< boost::char_separator<char> > stringTok2(std::string("A1B:12"),sep);
  boost::tokenizer< boost::char_separator<char> >::iterator tokIter2=stringTok2.begin();

  while(tokIter2!=stringTok2.end()) {
    std::cout << "Token: " << (*tokIter2) << "\n";
    tokIter2++;
  }
}
---Cut Here---

-- 
Thomas Johnson <thomas@unifiedconsulting.com>