RE: [Boost-users] Can boost::tokenizer tokenize 2byte character string?

From: Ben Hutchings (ben.hutchings_at_[hidden])
Date: 2004-09-20 11:55:13

Joo-Young Lee <trowind_at_[hidden]> wrote:
> I try to use 'boost::tokenizer<boost::char_separator<char> >' 
> to separate 2byte character string like Korean, Japanese, or
> Chinese.
> 
> But, I found that it does not works correctly.
> 
> Is there a solution?

You should probably convert them to wide strings and use the
wchar_t version instead.  The standard C++ library and Boost
libraries tend not to work well with multibyte encodings.