RE: [Boost-Users] tokenizer

From: Jim.Hyslop (jim.hyslop_at_[hidden])
Date: 2002-09-03 07:55:42

Vijay (macrodisk_at_[hidden]) wrote:
> Is there anyway I can combine two tokens and check it as single one?
> 
> I want to divide my string when both "-" and ">" occur together:
> 
> Cell 12-1 -> xyz...
> 
> I want to get the first token as Cell 12-1, how can I achive this via
> tokenizer?
I haven't used the tokenizer, but it seems to me that your TokenizerFunction
can peek at the character after the '-' to see if it is a '>', to determine
whether it should treat the '-' as a regular character or part of a
separator.

-- 
Jim