Re: [boost] Interest in a token iterator???

From: Aleksey Gurtovoy (alexy_at_[hidden])
Date: 2000-08-23 08:31:48

----- Original Message -----
From: <jbandela_at_[hidden]>
To: <boost_at_[hidden]>
Sent: Tuesday, August 22, 2000 1:04 PM
Subject: [boost] Interest in a token iterator???

> Is there any interest in a token iterator class? The class is
> templatized on the string type, the parsing functor, and - through
> the parsing functor - the actual type of the token.
>

I like the concept. However, your implementation is tied to strings (even
although StringType is a template parameter), which is too restrictive, IMO.
I would like to see a more generic interface, e.g. one that allows to create
a token_iterator for arbitrary input sequence; for instance, I would like to
be possible to write something like this:

  using namespace boost;
  using namespace std;

  string s( "..." );
  token_iterator<char, ...> itor( s.begin(), s.end() );
  token_iterator<char, ...> end;
  find( itor, end, "..." );
or
  char buf[256];
  token_iterator<char, ...> itor( buf, buf + sizeof(buf) );
  token_iterator<char, ...> end;
  find( itor, end, "..." );
or
  ifstream stream( "1.txt" );
  token_iterator<char, ...> itor( istream_iterator<char>( stream ),
    istream_iterator<char>() );
  token_iterator<char, ...> end;
  find( itor, end, "..." );

Unfortunately, implementing this (in particular, parameterization of
token_iterator by the type of the items in the original sequence, not by the
type of the sequence iterator) may be not very easy (or/and practical).
Probably I would be satisfied with something less challenging, but still
useful - so the following would work:

  string s( "..." );
  token_iterator<string::const_iterator> itor( s.begin(), s.end() );
  token_iterator<string::const_iterator> end;
  find( itor, end, "..." );
or
  ifstream stream( "1.txt" );
  token_iterator< istream_iterator<char> >
r( 
    istream_iterator<char>( stream ), 
    istream_iterator<char>() );
  token_iterator< istream_iterator<char> > end;
  find( itor, end, "..." );

Thoughts?

--Aleksey