Re: [boost] Tokenizer documentation?

From: David Abrahams (david.abrahams_at_[hidden])
Date: 2001-10-15 21:12:17

----- Original Message -----
From: "George A. Heintzelman" <georgeh_at_[hidden]>

> Second, and more crucial -- from hard-earned experience, and looking at
> the source, it appears that dereferencing a token_iterator returns a
> reference to an internal cache. This is fine, except that it means that:
>
> const string &ref = *tok_it++;
>
> does not work -- the token iterator overwrites the internal data, and
> ref now will contain whatever is in the second token. Since this
> doesn't happen for most iterators in the STL, I think this feature
> needs to be loudly documented.

As surprising as it may be, it is legal for forward iterators (and above) to
return a reference to an internally-stored object when dereferenced.

However, the expression tok_it++ is required to have the operational
semantics:

{ token_iterator x = tok_it; ++tok_it; return x }

So (if token_iterator is a conforming ForwardIterator) the token iterator
being dereferenced still contains the value of the original token.
Unfortunately, its lifetime ends at the end of the current expression, so
you can never use ref above: once the initialization is finished, the
temporary token_iterator is destroyed and ref is left dangling.

It is still possible that token_iterator is not conforming and instead does
something like what you say. In any case, I agree that the internal caching
behavior is unusual and should be documented.

> I'd actually prefer if it went away from
> the default policy implementation -- return the string by value if
> necessary -- but that's a larger change than documenting the existing
> behavior.

It would also limit token_iterator's category to InputIterator, since the
result of dereferencing a ForwardIterator is required to be a reference.

===================================================
  David Abrahams, C++ library designer for hire
 resume: http://users.rcn.com/abrahams/resume.html

        C++ Booster (http://www.boost.org)
          email: david.abrahams_at_[hidden]
===================================================