Boost logo

Boost :

From: jbandela_at_[hidden]
Date: 2000-08-24 18:22:16

As I read over your post, I find myself agreeing with you more. It
was not too hard to change it to iterators. In addition, the iterator
category is defined to be input_iterator for input iterators and
forward_iterator for all others.

I probably will be posting a new version for evaluation.

--- In boost_at_[hidden], "Aleksey Gurtovoy" <alexy_at_m...> wrote:
> ----- Original Message -----
> From: <jbandela_at_u...>
> To: <boost_at_[hidden]>
> Sent: Wednesday, August 23, 2000 9:32 AM
> Subject: [boost] Re: Interest in a token iterator???
> > I had thought about doing it. However, it seems the main benefit
> > would be when using istream_iterators.
> The main benefit is that with such interface you could use a pair of
> iterators of *an arbitrary type* to specify an input for your token
> iterator; you would not be forced to use 'std::string' or any other
> container-like object to specify just an input sequence to be
> IMO, pair of iterators is much more natural way to do it and it's
> generic.
> > Other than that, a char buffer
> > is easily converted to a string.
> Sometimes an overhead of that conversion (copying) may be
unacceptable or
> data may be just too big to place it in memory (and you don't want
> process it in chunks).
> > The problem with istream_iterators
> > is that they are input iterators. If you have two input iterators
> > referencing the same sequence, modifying one, modifies the other.
> > Thus, if token iterator was implemented in terms of an input
> > iterator. Having independent copies of the iterator that reference
> > the same character sequence would be impossible. This would mean
> > you could not pass token iterators into algorithms such as copy or
> > find, without having your original token iterators modified.
> Strictly speaking, that's not true ;) Your statement is correct
only for
> 'token_iterator< istream_iterator<...> >', or whatever other
> 'token_iterator<>' parameterized by an iterator type which
satisfies only
> input iterator requirements. But in that case you are getting what
> asked for.
> I think what you need is a proper definition of the concept, which
> that the iterator category of the token iterator depends on the
> category of iterators used to iterate through original input
sequence - e.g.
> 'token_iterator<std::string::const_iterator>::iterator_category'
can be
> 'std::forward_iterator_tag'.
> > This
> > could also seriously affect algorithms that depend on lookahead
> > features (ie = vs == in C/C++).
> If you accept my definition above, it will not (or it will only in
case if
> you want so :).
> > Finally, there is the ownership
> > issue. If the sequence is modified or deleted, the token iterators
> > could become corrupt.
> I don't think that's a problem. After all, if you modify or delete
a vector,
> all its iterators become invalid too :)
> > Based on all this, I decided to use strings
> > that the token iterator owned. In addition, since the string is
> > modified, the string is reference counted and shared between all
> > copies of a token iterator. This makes copying them pretty cheap.
> >
> Sorry, I don't like this. First, you create a copy of the StringType
> parameter on the heap, which may be quite expensive. Second, using a
> reference counting may be a show stopper for possible users of the
class who
> have to deal with multi-threading (it's not thread-safe, isn't
it? ;).
> Third, IMHO, the problems you were trying to solve might be not
indeed a
> problems, so we can get rid of all these complications if you agree
with my
> points.
> Does it make any sense to you?
> --Aleksey

Boost list run by bdawes at, gregod at, cpdaniel at, john at