Re: tokenizer profiling results

From: jbandela_at_[hidden]
Date: 2001-10-19 19:34:55

I have posted a test version of tokenizer to tokenizer_optimize.zip 
in the files section. Here are the pertinent changes with respect to 
your posts.

1) A single function object is shared across iterators that belong to 
a tokenizer

2)Any non-constant state is externalized for a function object

3)Although token_iterator_base still keeps a current iterator in 
order to allow all types to be used with tokenizer, it is possible to 
partial specialize token_iterator_base for even more efficiency (ie 
token_rep where the current iterator is the end iterator of the token)

When I tested token iterator for size, with std::string and STLPort, 
with char_delimiters_separator<char>, The size of the iterator was 
20, of which 12 was the string.

Let me know what you think,

Regards,

John R. Bandela