boost::tokenizer

From: peter (yg-boost-users_at_[hidden])
Date: 2003-08-04 20:03:13

hi there

im trying to tokenize a string where any sort of blank space (' ', or '\t')
is a delimiter, and allow having spaces in a token by enclosing it with a
single quote (') character...
the code below does not work well if i have more than one space characters
(' ' or '\t') beside eachother... it simply keeps the token (even though the
result is an empty string)...

#include <string>
#include <vector>
#include <boost/tokenizer.hpp>

std::vector<std::string> tokenizeStr(const std::string &str)
 {
  std::vector<std::string> result;

  boost::escaped_list_separator<char> sep("", " \t", "'");

  boost::tokenizer<boost::escaped_list_separator<char> > tok(str, sep);

  for(boost::tokenizer<boost::escaped_list_separator<char> >::iterator
p=tok.begin(); p!=tok.end(); p++)
  {
   result.push_back(static_cast<std::string>(*p));
  }
  return result;

 }

does anyone have any ideas how to make this work without actually going
through the list manually and deleting the empty tokens?? by the way, im
using boost 1.29

thanks
peter