[boost] [Boost-bugs] [ boost-Bugs-1003889 ] tokenizer and signed char

From: SourceForge.net (noreply_at_[hidden])
Date: 2004-08-05 06:41:13

Bugs item #1003889, was opened at 2004-08-05 04:41
Message generated for change (Tracker Item Submitted) made by Item Submitter
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=107586&aid=1003889&group_id=7586

Category: None
Group: None
Status: Open
Resolution: None
Priority: 5
Submitted By: Nobody/Anonymous (nobody)
Assigned to: Nobody/Anonymous (nobody)
Summary: tokenizer and signed char

Initial Comment:
boost::tokenizer incorrect works on platforms on which 
plain char is signed. For example, at least in 
boost::char_delimiters_separator::is_nonret should be 
explicit conversion from char type to int type. 
Something like this std::isspace
(std::char_traits<Char>::to_int_type(E)). Minimal 
reproducer for this issue which uses Russian letters 
results in assert in MSVC 7.1:

// cl /Zi /MDd -D_DEBUG test.cpp
#include <boost\tokenizer.hpp>
#include <string>
#include <iostream>
using namespace std;
using namespace boost;  
  
int main()
{
  string s = "&#1047;&#1076;&#1088;&#1072;&#1074;&#1089;&#1090;&#1074;&#1091;&#1081; &#1084;&#1080;&#1088;!";
    
  tokenizer<> tok(s);
  for (tokenizer<>::iterator beg=tok.begin(); beg!
=tok.end();++beg)
  {
    cout << *beg << "\n";
  }
}

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=107586&aid=1003889&group_id=7586

-------------------------------------------------------
This SF.Net email is sponsored by OSTG. Have you noticed the changes on
Linux.com, ITManagersJournal and NewsForge in the past few weeks? Now,
one more big change to announce. We are now OSTG- Open Source Technology
Group. Come see the changes on the new OSTG site. www.ostg.com
_______________________________________________
Boost-bugs mailing list
Boost-bugs_at_[hidden]
https://lists.sourceforge.net/lists/listinfo/boost-bugs