Boost logo

Boost :

From: jbandela_at_[hidden]
Date: 2000-08-23 09:32:05

I had thought about doing it. However, it seems the main benefit
would be when using istream_iterators. Other than that, a char buffer
is easily converted to a string. The problem with istream_iterators
is that they are input iterators. If you have two input iterators
referencing the same sequence, modifying one, modifies the other.
Thus, if token iterator was implemented in terms of an input
iterator. Having independent copies of the iterator that reference
the same character sequence would be impossible. This would mean that
you could not pass token iterators into algorithms such as copy or
find, without having your original token iterators modified. This
could also seriously affect algorithms that depend on lookahead
features (ie = vs == in C/C++). Finally, there is the ownership
issue. If the sequence is modified or deleted, the token iterators
could become corrupt. Based on all this, I decided to use strings
that the token iterator owned. In addition, since the string is never
modified, the string is reference counted and shared between all
copies of a token iterator. This makes copying them pretty cheap.

These are my current thoughts. If anyone can provide a way around
these issues, I would be most interested.

--- In boost_at_[hidden], "Aleksey Gurtovoy" <alexy_at_m...> wrote:
> ----- Original Message -----
> From: <jbandela_at_u...>
> To: <boost_at_[hidden]>
> Sent: Tuesday, August 22, 2000 1:04 PM
> Subject: [boost] Interest in a token iterator???
> > Is there any interest in a token iterator class? The class is
> > templatized on the string type, the parsing functor, and - through
> > the parsing functor - the actual type of the token.
> >
> I like the concept. However, your implementation is tied to strings
> although StringType is a template parameter), which is too
restrictive, IMO.
> I would like to see a more generic interface, e.g. one that allows
to create
> a token_iterator for arbitrary input sequence; for instance, I
would like to
> be possible to write something like this:
> using namespace boost;
> using namespace std;
> string s( "..." );
> token_iterator<char, ...> itor( s.begin(), s.end() );
> token_iterator<char, ...> end;
> find( itor, end, "..." );
> or
> char buf[256];
> token_iterator<char, ...> itor( buf, buf + sizeof(buf) );
> token_iterator<char, ...> end;
> find( itor, end, "..." );
> or
> ifstream stream( "1.txt" );
> token_iterator<char, ...> itor( istream_iterator<char>( stream ),
> istream_iterator<char>() );
> token_iterator<char, ...> end;
> find( itor, end, "..." );
> Unfortunately, implementing this (in particular, parameterization of
> token_iterator by the type of the items in the original sequence,
not by the
> type of the sequence iterator) may be not very easy (or/and
> Probably I would be satisfied with something less challenging, but
> useful - so the following would work:
> string s( "..." );
> token_iterator<string::const_iterator> itor( s.begin(), s.end() );
> token_iterator<string::const_iterator> end;
> find( itor, end, "..." );
> or
> ifstream stream( "1.txt" );
> token_iterator< istream_iterator<char> >
> r(
> istream_iterator<char>( stream ),
> istream_iterator<char>() );
> token_iterator< istream_iterator<char> > end;
> find( itor, end, "..." );
> Thoughts?
> --Aleksey

Boost list run by bdawes at, gregod at, cpdaniel at, john at