Re: Tokenizer and empty tokens

From: Gennadiy E. Rozental (rogeeff_at_[hidden])
Date: 2001-08-31 13:27:08

--- In boost_at_y..., jbandela_at_u... wrote:
> Hi Gennadiy
> 
> First, I would like some clarification. What do you mean by string 
> with const semantics?
You are using appending to polulate generated token:

   tok += *next;

It implicitly assumes that Token class will allocate memory to adopt 
new character(may be not with every append). Because you don't 
provide address Token class can't be implemented as pointer into 
source memory, something like this:

class const_string {
public:
    assign( const char* begin, const char* end ); 
    assign( const char* begin, size_t len ); 
private:
    const char* begin;
    const char* end;
};

No way this class can be used with append logic. And it works 
perfectly with assing - no additionla memory needed. Also I want to 
mention that assign will be a way more efficient for 
RandomAccessIterator.

> 
> The major reason, I did not do the assign(cur,next) was because it 
is 
> incompatible with with input iterators because it requires 2 passes 
> through the sequence - 1 to determine the end of the token, and 
> another to actually assign it.
I got your point. But majority of users will be using either 
string::iterator or char* as InputIterator (both 
RandomAccessIterators). The only concern could be with 
istream_iterator-like case. To accomodate it we should develop policy 
class. Default version will work efficiently using assign and 
InputIterator version will use append.

Regards,

Gennadiy.

> 
> Regards,
> 
> John R. Bandela
> 
> --- In boost_at_y..., "Gennadiy E. Rozental" <rogeeff_at_m...> wrote:
> > Hi, John
> > 
> > while you are looking into token_function I would like to remind 
my 
> > proposition to make following change:
> > 
> > Index: token_functions.hpp
> > 
===================================================================
> > RCS file: /cvsroot/boost/boost/boost/token_functions.hpp,v
> > retrieving revision 1.8
> > diff -r1.8 token_functions.hpp
> > 278c278
> > < 			 tok+=*next;
> > ---
> > > 			 tok.assign( next, 1 );
> > 281,284c281,287
> > < 		 else
> > < 			 // append all the non delim characters
> > < 			 for(;next!=end && !is_nonret(*next) && !
> > is_ret(*next);++next)
> > < 				 tok+=*next;
> > ---
> > >          else {
> > >              InputIterator curr = next;
> > > 			 // append all the non delim characters
> > > 			 for(;next!=end && !is_nonret(*next) && !
> > is_ret(*next);++next);
> > > 
> > >              tok.assign( curr, next );
> > >          }
> > ( The same can be done in most cases for offset separator 
function 
> > and can't be done for escaped_list_separator)
> > 
> > Reason: to allow using string class with const semantic.
> > 
> > What do you think about it?
> > 
> > Gennadiy.