[boost] Tokenizer: token_iterator undocumented members

From: Fredrik Blomqvist (fredrik_blomqvist_at_[hidden])
Date: 2004-01-13 18:36:52

Hi

While using the tokenizer I found that there are several members and
constructors that are not documented.

: token_iterator::
    base(), end(),
    tokenizer_function(), current_token(), at_end()

Is this just because out lagging documentation or are they intended to be
implementation details? Although not part of the "pure" std-iterator
interface they certainly seem useful from a user's perspective.

IMO the ability to extract the remaining part of the input using [it.base(),
it.end() ) is very useful functionality (which the Java tokenizer lacks for
example..) and would benefit from becoming official. ("Desperately" looking
for this feature was the reason I opened the .hpp in the first place ;-)

Also I find it a bit odd that token_iterator provides tokenizer_function()
but the tokenizer itself doesn't?

I couldn't find any mention of either of these issues in the mail-archives.

Regards
// Fredrik Blomqvist