RE: [Boost-users] Can boost::tokenizer tokenize 2byte character string?

From: Delfin Rojas (drojas_at_[hidden])
Date: 2004-09-20 13:10:47

Here you are:

typedef std::basic_string<TCHAR> tstring_t;
typedef boost::char_separator<TCHAR> Delimiter_t;
typedef boost::tokenizer< Delimiter_t, tstring_t::const_iterator, tstring_t
> TabTokenizer_t; 

Delimiter_t csDelimiter(#my_delimeter_tchar#);
TabTokenizer_t tTokens(#my_input_tchar_string#, csDelimiter);

TabTokenizer_t::iterator itTok(tTokens.begin());
TabTokenizer_t::iterator itTokEnd(tTokens.end());

while (itTok != itTokEnd)
{
        //do stuff
}
                  
This will tokenize an input tstring_t. I used TCHAR because they are
translated as char or wchar_t depending if Unicode is turned on or not. If
you are only interested in wide char strings then used wchar_t directly.

-delfin
 

> -----Original Message-----
> From: boost-users-bounces_at_[hidden] [mailto:boost-users-
> bounces_at_[hidden]] On Behalf Of Rozental, Gennadiy
> Sent: Monday, September 20, 2004 8:59 AM
> To: 'boost-users_at_[hidden]'
> Subject: RE: [Boost-users] Can boost::tokenizer tokenize 2byte character
> string?
> 
> If it 2byte why are you using boost::char_separator<char>?
> 
> Shouldn't you be using boost::char_separator<wchar_t>?
> 
> Gennadiy.
> 
> > -----Original Message-----
> > From: boost-users-bounces_at_[hidden]
> > [mailto:boost-users-bounces_at_[hidden]] On Behalf Of
> > Lee, Joo-Young
> > Sent: Saturday, September 18, 2004 12:39 AM
> > To: boost-users_at_[hidden]
> > Subject: [Boost-users] Can boost::tokenizer tokenize 2byte
> > character string?
> >
> >
> > Hi.
> >
> > I try to use 'boost::tokenizer<boost::char_separator<char> >'
> > to separate 2byte character string like Korean, Japanese, or Chinese.
> >
> > But, I found that it does not works correctly.
> >
> > Is there a solution?
> >
> > Thanks for the help,
> >
> > Lee Joo-Young
> >
> >
> >
> > _______________________________________________
> > Boost-users mailing list
> > Boost-users_at_[hidden]
> > http://lists.boost.org/mailman/listinfo.cgi/boost-users
> >
> _______________________________________________
> Boost-users mailing list
> Boost-users_at_[hidden]
> http://lists.boost.org/mailman/listinfo.cgi/boost-users