[boost] [tokenizer and token_iterator] request for policy or redesign

From: Thorsten Ottosen (nesotto_at_[hidden])
Date: 2004-09-28 13:03:41

Hi John,

I vaguely recall that the standard committee rejected the tokenizer proposal
because of performance problems.
A thing that would boost performance a lot would be to allow the user to
specify a "token match type".

So I have some stupid questions:

1. Why is the three template paramters not defined as

template <
        class TokenizerFunc = char_delimiters_separator<char>,
class Type = std::string
class Iterator = typename range_const_iterator<Type>::type
  >
  class tokenizer

??

2. what is the requirement exactly on the Type parameter?

3. would it not be possible to allow

typedef boost::tokenizer< fun, boost::sub_range<string> > tokenizer;

? I believe this would give a huge speedup.

br

Thorsten