Re: Tokenizer and empty tokens

From: jbandela_at_[hidden]
Date: 2001-08-31 10:58:13

Hi Gennadiy

First, I would like some clarification. What do you mean by string 
with const semantics?

The major reason, I did not do the assign(cur,next) was because it is 
incompatible with with input iterators because it requires 2 passes 
through the sequence - 1 to determine the end of the token, and 
another to actually assign it.

Regards,

John R. Bandela

--- In boost_at_y..., "Gennadiy E. Rozental" <rogeeff_at_m...> wrote:
> Hi, John
> 
> while you are looking into token_function I would like to remind my 
> proposition to make following change:
> 
> Index: token_functions.hpp
> ===================================================================
> RCS file: /cvsroot/boost/boost/boost/token_functions.hpp,v
> retrieving revision 1.8
> diff -r1.8 token_functions.hpp
> 278c278
> < 			 tok+=*next;
> ---
> > 			 tok.assign( next, 1 );
> 281,284c281,287
> < 		 else
> < 			 // append all the non delim characters
> < 			 for(;next!=end && !is_nonret(*next) && !
> is_ret(*next);++next)
> < 				 tok+=*next;
> ---
> >          else {
> >              InputIterator curr = next;
> > 			 // append all the non delim characters
> > 			 for(;next!=end && !is_nonret(*next) && !
> is_ret(*next);++next);
> > 
> >              tok.assign( curr, next );
> >          }
> ( The same can be done in most cases for offset separator function 
> and can't be done for escaped_list_separator)
> 
> Reason: to allow using string class with const semantic.
> 
> What do you think about it?
> 
> Gennadiy.