Re: [Boost-Users] boost tokenizer

From: Jeremy Siek (jsiek_at_[hidden])
Date: 2001-11-23 15:53:52

Hi Paul,

By default the char_delimiters_separator (which is the default
TokenizerFunction for token_iterator and tokenizer) splits on any
punctuation, which I assume includes '_'. To explicitly specify the
splitting characters, you pass a string of the separator characters into the
constructor, like this:

char_delimiters_separator<char> sep(false, " |");
// create token iterator or tokenizer using sep

Cheers,
Jeremy

On 11/23/01 10:21 AM, "pjgrenyer_at_[hidden]" <pjgrenyer_at_[hidden]> wrote:

> Dear All
> 
> I'm using the boost tokenizer to tokenise lines similar to this one:
> 
> first_initial        |23|1|
> 
> This gives me:
> 
> first
> initial
> 23
> 1
> 
> but what I actually want is:
> 
> first_initial
> 23
> 1
> 
> Does anyone know how to get it to ignore the underscore?
> 
> Thanks
> Paul

-- 
Jeremy Siek                          http://www.osl.iu.edu/~jsiek
Ph.D. Student, Indiana Univ. B'ton   email: jsiek_at_[hidden]
C++ Booster (http://www.boost.org)   office phone: [phone]