At 19:19 2005-06-11, you wrote:

> for tokenizing on whitespace, simple stream input (>>) to a
> std::string suffices.

My own tokenizer does just that--and puts the tokens into a deque.

> IMO, it's hardly worth troubling yourself with a tokenizer
> for whitespace.

Well, not really. When parsing line-oriented output and semi-known
structured lines it's handy to be able to sometimes work with a line's
tokens as if they were in a vector or deque.

        string yourline;
        istringstream is( yourline );
        deque < string > yourvec(( istream_iterator < std :: string >( is )), istream_iterator <std :: string >());

voila, a deque

it would be interesting to profile that against the hypothetical indexable tokenizer.

In fact, I was going to add a suggestion that the tokenizer also have the []
operator so that the individual tokens could be addressed as tok[1], etc.

-Tom

_______________________________________________
Boost-users mailing list
Boost-users@lists.boost.org
http://lists.boost.org/mailman/listinfo.cgi/boost-users

Victor A. Wagner Jr. http://rudbek.com
The five most dangerous words in the English language:
"There oughta be a law"