token iterator

From: Jeremy Siek (jsiek_at_[hidden])
Date: 2000-12-29 00:38:49

Starting with John's token iterators, I've created a new version.
It's in the egroups files: TokenIterator/tokenizer.zip

Similarities:
 - I kept the Tokenizer function idea, renamed it TokenizerFunction 
 - I kept the basic idea of using an iterator interface

Differences:
 - The token iterator is simplified to *just* the iterator interface,
    no extra functions.
 - A token "container-view" is provided to simplify the creation
    of begin/end pairs of iterator.
 - Added a tokenizer function that is a generalized form of strtok().

Here's an example

  boost::array<std::string, 3> separator_list = { { ",", ":", "--" } };
  std::string input_string = "::blah--blat:bloop,barf----";

  // create the tokenizer function
  boost::token_separator<char> t1(separator_list);

  // create the token container-view, using the tokenizer class
  typedef boost::tokenizer<std::string::iterator, 
    boost::token_separator<char>, std::string > TokenContainer;
  TokenContainer tokens(input_string.begin(), input_string.end(), t1);

  // Parse the input string, priting each token
  for (TokenContainer::iterator i = tokens.begin();
       i != tokens.end(); ++i)
    std::cout << "\"" << *i << "\" ";
  std::cout << std::endl;

Cheers,
Jeremy

----------------------------------------------------------------------
 Jeremy Siek                        www: http://www.lsc.nd.edu/~jsiek/
 Ph.D. Candidate                  email: jsiek_at_[hidden]
 Univ. of Notre Dame         work phone: [phone]
----------------------------------------------------------------------