RE: [boost] Re: Tokenizer and empty tokens

From: johan.nilsson_at_[hidden]
Date: 2001-09-07 03:09:41

Thanks,

I've been off for a while now. I'll look into it and try it out asap.

// Johan

> -----Original Message-----
> From: jbandela_at_[hidden] [mailto:jbandela_at_[hidden]]
> Sent: den 31 augusti 2001 20:22
> To: boost_at_[hidden]
> Subject: [boost] Re: Tokenizer and empty tokens
> 
> 
> Johan 
> Updated 
> http://groups.yahoo.com/group/boost/files/token_functions.hpp
> 
> I am now thinking that char_delimiters_separator is not the place to 
> do empty tokens. Since emtpy tokens almost always involve fields, I 
> think escaped_list_separator is a better place for it. To support 
> problems like yours I did 2 things
> 
> 1. Made sure it returned trailing empty tokens
> 2. Allow you to specify multiple characters as escape,c,quote
> 
> Here is an example
> 
>   string s = "$GPGSA,A,3,25,04,16,06,18,24,,,,,,,1.86,0.92,1.62*0E,";
>   typedef escaped_list_separator<char> sep_type;
>   sep_type  sep("","$,*","");
>   tokenizer<sep_type> t(s,sep);
>   copy(t.begin(),t.end(),ostream_iterator<string>(cout,"\n"));
> 
> Regards
> 
> John R. Bandela
> 
> 
> --- In boost_at_y..., johan.nilsson_at_e... wrote:
> > First of all: yes, now it works in my particular case.
> > 
> > Couple of notes though:
> > 
> > * [when allowing empty tokens] char_delimiters_separator outputs 
> leading,
> > but not trailing empty tokens, e.g.:
> > 
> > tokenizing ",test," outputs ("" "test") instead of ("" "test" ""), 
> which at
> > least I would expected.
> > 
> > * Off the top of my head, it seems like escaped_list_separator 
> should be
> > able to do the same thing (output empty tokens). Which leads to the 
> question
> > on _where_ the policy on outputting empty tokens should be adapted, 
> or?
> > 
> > // Johan
> > 
> 
> 
> 
> Info: http://www.boost.org  Unsubscribe: 
<mailto:boost-unsubscribe_at_[hidden]> 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/