boost::tokenizer problem, maybe a bug.

Hi. I discovered the following problem using boost::tokenizer. The following program should print "hello", but it prints "hell": #include <iostream> #include <boost/tokenizer.hpp> int main(){ typedef boost::tokenizer<boost::char_separator<char> > tokenizer; boost::char_separator<char> sep(",", ""); tokenizer values(std::string(",hello,"), sep); for( tokenizer::iterator valueI = values.begin(); valueI != values.end(); ++valueI){ std::cout << "\"" << *valueI << "\"" << std::endl; } } Problem is confirmed using boost 1.32.0 and both GCC 3.3.4 and GCC 3.3.5 If anybody have a newer GCC installed, I would like to hear if the problems also exists in later GCC releases. I believe it is a bug in the std::string::assign implementation (i.e. not boost directly), but a the workaround for msvc in boost/token_functions.hpp, line 215-228 has also been tried for the GCC but it just moves the problem; it will mibehave when the line tokenizer values(std::string(",hello,"), sep); is replaced with tokenizer values(std::string("hello,"), sep); since it will then print "hel". So maybe another boost workaround is needed for GCC for the lines 215-228 in boost/token_functions.hpp Jarl -- Jarl Friis Softace ApS Omøgade 8, 2.sal 2100 København Ø. Denmark Phone: +45 26 13 20 90 E-mail: jarl@softace.dk

Jarl Friis wrote:
Hi.
I discovered the following problem using boost::tokenizer. The following program should print "hello", but it prints "hell":
#include <iostream> #include <boost/tokenizer.hpp>
int main(){ typedef boost::tokenizer<boost::char_separator<char> > tokenizer;
boost::char_separator<char> sep(",", ""); tokenizer values(std::string(",hello,"), sep);
Your string is a temporary and goes out of scope. Try: std::string test(",hello,"); tokenizer values( test, sep );
for( tokenizer::iterator valueI = values.begin(); valueI != values.end(); ++valueI){ std::cout << "\"" << *valueI << "\"" << std::endl; } }
Jeff

Jeff Flinn wrote:
Jarl Friis wrote:
Hi.
I discovered the following problem using boost::tokenizer. The following program should print "hello", but it prints "hell":
#include <iostream> #include <boost/tokenizer.hpp>
int main(){ typedef boost::tokenizer<boost::char_separator<char> > tokenizer;
boost::char_separator<char> sep(",", ""); tokenizer values(std::string(",hello,"), sep);
Your string is a temporary and goes out of scope. Try:
std::string test(",hello,");
tokenizer values( test, sep );
hm, why thiis works with vc-7_1 then? ;)
for( tokenizer::iterator valueI = values.begin(); valueI != values.end(); ++valueI){ std::cout << "\"" << *valueI << "\"" << std::endl; } }
-- Serge

Serge Skorokhodov <serge.skorokhodov@tochka.ru> writes:
Jeff Flinn wrote:
Jarl Friis wrote:
Hi.
I discovered the following problem using boost::tokenizer. The following program should print "hello", but it prints "hell":
#include <iostream> #include <boost/tokenizer.hpp>
int main(){ typedef boost::tokenizer<boost::char_separator<char> > tokenizer;
boost::char_separator<char> sep(",", ""); tokenizer values(std::string(",hello,"), sep);
Your string is a temporary and goes out of scope. Try:
std::string test(",hello,");
tokenizer values( test, sep );
Thanks, that helped.
hm, why thiis works with vc-7_1 then? ;)
I gues accoriding to the standard, it is undefined behaviour, so it might work, you just can't rely on it. Jarl
participants (3)
-
Jarl Friis
-
Jeff Flinn
-
Serge Skorokhodov