RE: [boost] Tokenizer and empty tokens

From: johan.nilsson_at_[hidden]
Date: 2001-08-30 02:55:47

First of all: yes, now it works in my particular case.

Couple of notes though:

* [when allowing empty tokens] char_delimiters_separator outputs leading,
but not trailing empty tokens, e.g.:

tokenizing ",test," outputs ("" "test") instead of ("" "test" ""), which at
least I would expected.

* Off the top of my head, it seems like escaped_list_separator should be
able to do the same thing (output empty tokens). Which leads to the question
on _where_ the policy on outputting empty tokens should be adapted, or?

// Johan

> -----Original Message-----
> From: jbandela_at_[hidden] [mailto:jbandela_at_[hidden]]
> Sent: den 29 augusti 2001 19:31
> To: boost_at_[hidden]
> Subject: [boost] Tokenizer and empty tokens
> 
> 
> http://groups.yahoo.com/group/boost/files/token_functions.hpp
> 
> 
> Here is a version of token_functions that has a preliminary fix for 
> empty tokens for char_delimiters_separator. I have not yet put it 
> into the boost cvs because I would like to get more feedback first.
> 
> To get empty tokens, use the the following constructor
> 
> explicit char_delimiters_separator(const Char * returnable, bool 
> return_delims=false)
> 
> Here is a quick example
> 
> 	string s = ",how,, are,,you, doing";
> 	char_delimiters_separator<char> sep(",",false);
> 	tokenizer<> t(s,sep);
> 	copy(t.begin(),t.end(),ostream_iterator<string>(cout,"\n"));
> 
> 
> 
> Let me know what you think, both about the implementation, and the 
> interface.
> 
> Note, when returning empty tokens, all separators are returnable.
> 
> 
> Regards,
> 
> John R. Bandela
> 
> 
> 
> 
> 
> Info: http://www.boost.org  Unsubscribe: 
> <mailto:boost-unsubscribe_at_[hidden]> 
> 
> Your use of Yahoo! Groups is subject to 
> http://docs.yahoo.com/info/terms/ 
> 
>