[Boost-users] boost::tokenizer problem, maybe a bug.

From: Jarl Friis (jarl_at_[hidden])
Date: 2005-07-26 02:58:33

Hi.

I discovered the following problem using boost::tokenizer. The
following program should print "hello", but it prints "hell":

#include <iostream>
#include <boost/tokenizer.hpp>

int main(){
  typedef boost::tokenizer<boost::char_separator<char> > tokenizer;

  boost::char_separator<char> sep(",", "");
  tokenizer values(std::string(",hello,"), sep);
  for( tokenizer::iterator valueI = values.begin(); 
       valueI != values.end();
       ++valueI){
    std::cout << "\"" << *valueI << "\"" << std::endl;
  }
}

Problem is confirmed using boost 1.32.0 and both
GCC 3.3.4 and GCC 3.3.5

If anybody have a newer GCC installed, I would like to hear if the
problems also exists in later GCC releases.

I believe it is a bug in the std::string::assign implementation
(i.e. not boost directly), but a the workaround for msvc in
boost/token_functions.hpp, line 215-228 has also been tried for the
GCC but it just moves the problem; it will mibehave when the line
tokenizer values(std::string(",hello,"), sep); is replaced with
tokenizer values(std::string("hello,"), sep); since it will then print
"hel". So maybe another boost workaround is needed for GCC for the
lines 215-228 in boost/token_functions.hpp

Jarl

-- 
Jarl Friis
Softace ApS
Omøgade 8, 2.sal
2100 København Ø.
Denmark
Phone:  [phone]
E-mail: jarl_at_[hidden]