RE: [Boost-users] Can boost::tokenizer tokenize 2byte character string?

From: Ben Hutchings (ben.hutchings_at_[hidden])
Date: 2004-09-20 12:28:28

Gennadiy Rozental <gennadiy.rozental_at_[hidden]> wrote:
> If it 2byte why are you using boost::char_separator<char>? 
> 
> Shouldn't you be using boost::char_separator<wchar_t>?

I think Lee Joo-Young is talking about multibyte strings in an
encoding such as GB2312 and Shift_JIS that uses 1 or 2 bytes per
character rather than wide strings that use a fixed 2 bytes per
character on certain platforms (or rather, used to until UTF-16
came along).

Ben.