Re: [Boost-Users] boost tokenizer

From: Paul Grenyer (pjgrenyer_at_[hidden])
Date: 2001-11-23 16:33:22

Jeremy

Many thanks for this!

Paul

Paul Grenyer
pjgrenyer_at_[hidden]

  ----- Original Message ----- 
  From: Jeremy Siek 
  To: Boost-Users_at_[hidden] 
  Sent: Friday, November 23, 2001 8:53 PM
  Subject: Re: [Boost-Users] boost tokenizer

  Hi Paul,

  By default the char_delimiters_separator (which is the default
  TokenizerFunction for token_iterator and tokenizer) splits on any
  punctuation, which I assume includes '_'. To explicitly specify the
  splitting characters, you pass a string of the separator characters into the
  constructor, like this:

  char_delimiters_separator<char> sep(false, " |");
  // create token iterator or tokenizer using sep

  Cheers,
  Jeremy

  On 11/23/01 10:21 AM, "pjgrenyer_at_[hidden]" <pjgrenyer_at_[hidden]> wrote:

  > Dear All
  > 
  > I'm using the boost tokenizer to tokenise lines similar to this one:
  > 
  > first_initial        |23|1|
  > 
  > This gives me:
  > 
  > first
  > initial
  > 23
  > 1
  > 
  > but what I actually want is:
  > 
  > first_initial
  > 23
  > 1
  > 
  > Does anyone know how to get it to ignore the underscore?
  > 
  > Thanks
  > Paul

  -- 
  Jeremy Siek                          http://www.osl.iu.edu/~jsiek
  Ph.D. Student, Indiana Univ. B'ton   email: jsiek_at_[hidden]
  C++ Booster (http://www.boost.org)   office phone: [phone]

        Yahoo! Groups Sponsor 
       
       

  Info: <http://www.boost.org>
  Wiki: <http://www.crystalclearsoftware.com/cgi-bin/boost_wiki/wiki.pl>
  Unsubscribe: <mailto:boost-users-unsubscribe_at_[hidden]>

  Your use of Yahoo! Groups is subject to the Yahoo! Terms of Service. 

[Non-text portions of this message have been removed]