From: John Torjo (john.lists_at_[hidden])
Date: 2003-12-08 19:06:28
> One of the nice features of the current Tokenizerfunction concept is
> that it is a single pass algorithm, and will work with input
> iterators. I'm not sure how to keep the algorithm single pass if you
> split the token delimitation from the token creation.
The algorithm that I made is single-pass as well.
The implementation is already done. I want to do something else as well, and
then I'll post the rtl library again.
Basically, I realized the importance of single-pass. Therefore, there are two
types of slicers (or tokenizers, if you like):
- slice_byeach (which is single pass)
- slice_byrng (which allows operating on a sub-range, in order to return an
Also, note another post a few days ago, about making the tokenizer faster. As
I've seen, you've already posted an implementation for better Tokerizers. So,
watch out for mine ;)
Boost list run by bdawes at acm.org, gregod at cs.rpi.edu, cpdaniel at pacbell.net, john at johnmaddock.co.uk