Re: [boost] Re: A better tokenizer?

From: John Torjo (john.lists_at_[hidden])
Date: 2003-12-08 19:06:28

> 
> 
> One of the nice features of the current Tokenizerfunction concept is
> that it is a single pass algorithm, and will work with input
> iterators.  I'm not sure how to keep the algorithm single pass if you
> split the token delimitation from the token creation.
> 

The algorithm that I made is single-pass as well.
The implementation is already done. I want to do something else as well, and 
then I'll post the rtl library again.

Basically, I realized the importance of single-pass. Therefore, there are two 
types of slicers (or tokenizers, if you like):
- slice_byeach (which is single pass)
- slice_byrng (which allows operating on a sub-range, in order to return an 
element).

Also, note another post a few days ago, about making the tokenizer faster. As 
I've seen, you've already posted an implementation for better Tokerizers. So, 
watch out for mine ;)

Best,
John