Re: boost::tokenizer

From: peter (yg-boost-users_at_[hidden])
Date: 2003-08-05 02:21:29

"peter" <yg-boost-users_at_[hidden]> wrote in message
news:bgmvnl$e9p$1_at_main.gmane.org...
> hi there
>
> im trying to tokenize a string where any sort of blank space (' ', or
'\t')
> is a delimiter, and allow having spaces in a token by enclosing it with a
> single quote (') character...
> the code below does not work well if i have more than one space characters
> (' ' or '\t') beside eachother... it simply keeps the token (even though
the
> result is an empty string)...
>
> #include <string>
> #include <vector>
> #include <boost/tokenizer.hpp>
>
> std::vector<std::string> tokenizeStr(const std::string &str)
>  {
>   std::vector<std::string> result;
>
>   boost::escaped_list_separator<char> sep("", " \t", "'");
>
>   boost::tokenizer<boost::escaped_list_separator<char> > tok(str, sep);
>
>   for(boost::tokenizer<boost::escaped_list_separator<char> >::iterator
> p=tok.begin(); p!=tok.end(); p++)
>   {
>    result.push_back(static_cast<std::string>(*p));
>   }
>   return result;
>
>  }
>
> does anyone have any ideas how to make this work without actually going
> through the list manually and deleting the empty tokens?? by the way, im
> using boost 1.29
>
> thanks
> peter
>

i guess,  the trivial solution is just to check whether the string is
not-empty before adding it to 'result'

ie:
if (static_cast<std::string>(*p) != "")
       result.push_back(static_cast<std::string>(*p));

and that's what i am using now, but if someone knows of a way to have the
tokenizer disregard empty tokens automatically, please let me know...

cheers.