Re: Tokenizer and empty tokens

From: Gennadiy E. Rozental (rogeeff_at_[hidden])
Date: 2001-08-31 09:11:26

Hi, John

while you are looking into token_function I would like to remind my 
proposition to make following change:

Index: token_functions.hpp
===================================================================
RCS file: /cvsroot/boost/boost/boost/token_functions.hpp,v
retrieving revision 1.8
diff -r1.8 token_functions.hpp
278c278
< 			 tok+=*next;
---
> 			 tok.assign( next, 1 );
281,284c281,287
< 		 else
< 			 // append all the non delim characters
< 			 for(;next!=end && !is_nonret(*next) && !
is_ret(*next);++next)
< 				 tok+=*next;
---
>          else {
>              InputIterator curr = next;
> 			 // append all the non delim characters
> 			 for(;next!=end && !is_nonret(*next) && !
is_ret(*next);++next);
> 
>              tok.assign( curr, next );
>          }
( The same can be done in most cases for offset separator function 
and can't be done for escaped_list_separator)
Reason: to allow using string class with const semantic.
What do you think about it?
Gennadiy.
--- In boost_at_y..., jbandela_at_u... wrote:
> http://groups.yahoo.com/group/boost/files/token_functions.hpp
> 
> 
> Here is a version of token_functions that has a preliminary fix for 
> empty tokens for char_delimiters_separator. I have not yet put it 
> into the boost cvs because I would like to get more feedback first.
> 
> To get empty tokens, use the the following constructor
> 
> explicit char_delimiters_separator(const Char * returnable, bool 
> return_delims=false)
> 
> Here is a quick example
> 
> 	string s = ",how,, are,,you, doing";
> 	char_delimiters_separator<char> sep(",",false);
> 	tokenizer<> t(s,sep);
> 	copy(t.begin(),t.end(),ostream_iterator<string>(cout,"\n"));
> 
> 
> 
> Let me know what you think, both about the implementation, and the 
> interface.
> 
> Note, when returning empty tokens, all separators are returnable.
> 
> 
> Regards,
> 
> John R. Bandela