[Boost-users] [boost][tokenizer] std::wstring question

From: David Klein (d.klein_at_[hidden])
Date: 2005-06-08 04:27:36

hi,

Is there a reason why the template params for the tokenizer class are
std::string?
This makes it "harder" to use it with std::wstring.

You have to write:
        tokenizer<char_separator<wchar_t>, std::wstring::const_iterator,
std::wstring> tkz;
        ...

Instead of:
        tokenizer<char_separator<wchar_t> > tkz;
        ...

(i guess i'm very lazy ;)

I changed the scope of the string_type typedef for the xxx_separator
classes to public,
and used this typedef in the tokenizer template params.
Now std::wstring works like std::string for me.

*** token_functions.hpp ***

class char_separator/char_delimiters_separator/...
{
public:
        typedef std::basic_string<Char,Traits> string_type;
...
};

*** tokenizer.hpp ***

template <
        typename TokenizerFunc = char_delimiters_separator<char>, 
        typename Iterator = /*std::string::const_iterator*/
TokenizerFunc::string_type::const_iterator,
        typename Type = /*std::string*/ TokenizerFunc::string_type
>
class tokenizer {...};

regards
/dave