Re: Interest in a token iterator???

From: jbandela_at_[hidden]
Date: 2000-08-23 09:32:05

I had thought about doing it. However, it seems the main benefit 
would be when using istream_iterators. Other than that, a char buffer 
is easily converted to a string. The problem with istream_iterators 
is that they are input iterators. If you have two input iterators 
referencing the same sequence, modifying one, modifies the other. 
Thus, if token iterator was implemented in terms of an input 
iterator. Having independent copies of the iterator that reference 
the same character sequence would be impossible. This would mean that 
you could not pass token iterators into algorithms such as copy or 
find, without having your original token iterators modified. This 
could also seriously affect algorithms that depend on lookahead 
features (ie = vs == in C/C++). Finally, there is the ownership 
issue. If the sequence is modified or deleted, the token iterators 
could become corrupt. Based on all this, I decided to use strings 
that the token iterator owned. In addition, since the string is never 
modified, the string is reference counted and shared between all 
copies of a token iterator. This makes copying them pretty cheap.

These are my current thoughts. If anyone can provide a way around 
these issues, I would be most interested.

--- In boost_at_[hidden], "Aleksey Gurtovoy" <alexy_at_m...> wrote:
> 
> ----- Original Message -----
> From: <jbandela_at_u...>
> To: <boost_at_[hidden]>
> Sent: Tuesday, August 22, 2000 1:04 PM
> Subject: [boost] Interest in a token iterator???
> 
> 
> > Is there any interest in a token iterator class? The class is
> > templatized on the string type, the parsing functor, and - through
> > the parsing functor - the actual type of the token.
> >
> 
> I like the concept. However, your implementation is tied to strings 
(even
> although StringType is a template parameter), which is too 
restrictive, IMO.
> I would like to see a more generic interface, e.g. one that allows 
to create
> a token_iterator for arbitrary input sequence; for instance, I 
would like to
> be possible to write something like this:
> 
>   using namespace boost;
>   using namespace std;
> 
>   string s( "..." );
>   token_iterator<char, ...> itor( s.begin(), s.end() );
>   token_iterator<char, ...> end;
>   find( itor, end, "..." );
> or
>   char buf[256];
>   token_iterator<char, ...> itor( buf, buf + sizeof(buf) );
>   token_iterator<char, ...> end;
>   find( itor, end, "..." );
> or
>   ifstream stream( "1.txt" );
>   token_iterator<char, ...> itor( istream_iterator<char>( stream ),
>     istream_iterator<char>() );
>   token_iterator<char, ...> end;
>   find( itor, end, "..." );
> 
> Unfortunately, implementing this (in particular, parameterization of
> token_iterator by the type of the items in the original sequence, 
not by the
> type of the sequence iterator) may be not very easy (or/and 
practical).
> Probably I would be satisfied with something less challenging, but 
still
> useful - so the following would work:
> 
>   string s( "..." );
>   token_iterator<string::const_iterator> itor( s.begin(), s.end() );
>   token_iterator<string::const_iterator> end;
>   find( itor, end, "..." );
> or
>   ifstream stream( "1.txt" );
>   token_iterator< istream_iterator<char> >
> r( 
>     istream_iterator<char>( stream ), 
>     istream_iterator<char>() );
>   token_iterator< istream_iterator<char> > end;
>   find( itor, end, "..." );
> 
> Thoughts?
> 
> --Aleksey