Re: Tokenizer and empty tokens

From: jbandela_at_[hidden]
Date: 2001-08-31 13:22:12

Johan 
Updated 
http://groups.yahoo.com/group/boost/files/token_functions.hpp

I am now thinking that char_delimiters_separator is not the place to 
do empty tokens. Since emtpy tokens almost always involve fields, I 
think escaped_list_separator is a better place for it. To support 
problems like yours I did 2 things

1. Made sure it returned trailing empty tokens
2. Allow you to specify multiple characters as escape,c,quote

Here is an example

  string s = "$GPGSA,A,3,25,04,16,06,18,24,,,,,,,1.86,0.92,1.62*0E,";
  typedef escaped_list_separator<char> sep_type;
  sep_type  sep("","$,*","");
  tokenizer<sep_type> t(s,sep);
  copy(t.begin(),t.end(),ostream_iterator<string>(cout,"\n"));

Regards

John R. Bandela

--- In boost_at_y..., johan.nilsson_at_e... wrote:
> First of all: yes, now it works in my particular case.
> 
> Couple of notes though:
> 
> * [when allowing empty tokens] char_delimiters_separator outputs 
leading,
> but not trailing empty tokens, e.g.:
> 
> tokenizing ",test," outputs ("" "test") instead of ("" "test" ""), 
which at
> least I would expected.
> 
> * Off the top of my head, it seems like escaped_list_separator 
should be
> able to do the same thing (output empty tokens). Which leads to the 
question
> on _where_ the policy on outputting empty tokens should be adapted, 
or?
> 
> // Johan
>