RE: [Boost-users] Tokenizer Question

From: Tom Browder (tbrowder_at_[hidden])
Date: 2005-06-11 21:19:57

> for tokenizing on whitespace, simple stream input (>>) to a 
> std::string suffices.

My own tokenizer does just that--and puts the tokens into a deque.

> IMO, it's hardly worth troubling yourself with a tokenizer 
> for whitespace.

Well, not really.  When parsing line-oriented output and semi-known
structured lines it's handy to be able to sometimes work with a line's
tokens as if they were in a vector or deque.

In fact, I was going to add a suggestion that the tokenizer also have the []
operator so that the individual tokens could be addressed as tok[1], etc.

-Tom