Re: [Boost-users] Tokenizer Question

From: Martin (adrianm_at_[hidden])
Date: 2005-06-12 14:35:46

> My own tokenizer does just that--and puts the tokens into a
> deque.

Have you checked the string_algo library, specially the split function.

string str1("hello abc-*-ABC-*-aBc goodbye");

// ranges with iterators in original string

vector<iterator_range<string::iterator> > splitref;
split(splitref, str1, is_any_of("*") ); // splitref == { "hello abc-","-ABC-
","-aBc goodbye" }

// copies of found tokens

list<string> splitcopy;
split(splitcopy, str1, is_any_of("-*"), token_compress_on ); // splitcopy == 
{ "hello abc","ABC","aBc goodbye" }