[Boost-bugs] [Boost C++ Libraries] #1755: Boost.tokenizer is broken for wide characters - patch attached

Subject: [Boost-bugs] [Boost C++ Libraries] #1755: Boost.tokenizer is broken for wide characters - patch attached
From: Boost C++ Libraries (noreply_at_[hidden])
Date: 2008-04-02 14:33:13

#1755: Boost.tokenizer is broken for wide characters - patch attached
---------------------------------------+------------------------------------
 Reporter:  amit_at_[hidden]              |       Owner:  jsiek    
     Type:  Bugs                       |      Status:  new      
Milestone:  Boost 1.36.0               |   Component:  tokenizer
  Version:  Boost 1.35.0               |    Severity:  Problem  
 Keywords:  wchar_t, isspace, ispunct  |  
---------------------------------------+------------------------------------
 bug in token_functions.hpp - char_separator::is_kept and is_dropped call
 std::isspace and std::isdropped. But these cause an assert failure in
 debug build when parsing a wstring (at least on MSVC 7.1) since the CRT
 function isspace etc expect input to lie in the range 0-0xFF. MSDN
 documentation says that the behaviour is undefined outside this range;
 iswspace() etc are provided for use with wide characters.
 Please see the attached path for a fix to the problem.
 Note that this problem persists since at least 1.33, maybe even before
 then.

-- 
Ticket URL: <http://svn.boost.org/trac/boost/ticket/1755>
Boost C++ Libraries <http://www.boost.org/>
Boost provides free peer-reviewed portable C++ source libraries.