Boost logo

Boost Users :

From: Fredrik Hedman (hedman_at_[hidden])
Date: 2005-10-19 03:27:53


Hello,

it seems to me that boost::char_separator needs to have another ctor that can
accept delimiters that are string types. For example, given a
std::string("X\0Y\0\0Z", 6), it does not seem to be possible to use the
current ctor of boost::char_separator so that '\0' can be used as a
separator.

It is possible to use boost::escaped_list_separator, since *it* takes a string
type on construction, but on the other hand boost::escaped_list_separator
does not have an empty_token_policy. In summary, I suggest adding another
ctor to boost::char_separator. This will enable the parsing of the above
string as

   <X><Y><><Z>

or

   <X><Y><Z>

by the following:

#include <boost/tokenizer.hpp>
#include <iostream>
#include <string>

using namespace std;
using namespace boost;

int main()
{
  typedef tokenizer<boost::char_separator<char> > Tok;
  string str=string("X\0Y\0\0Z", 6);

  {
    char_separator<char> keep_empty(string(1,'\0'), string(),
                                    boost::keep_empty_tokens);
    Tok tokens(str, keep_empty);
    for(Tok::iterator iter = tokens.begin(); iter != tokens.end(); ++iter)
      cout << '<' << *iter << '>';
    cout << endl;
  }
  {
    char_separator<char> skip_empty(string(1,'\0'), string(),
                                    boost::drop_empty_tokens);
    Tok tokens(str, skip_empty);
    for(Tok::iterator iter = tokens.begin(); iter != tokens.end(); ++iter)
      cout << '<' << *iter << '>';
    cout << endl;
  }
}

This is achieved by a small change to class char_separator in
token_functions.hpp included in the attachment.

--
Best Regards,
Fredrik Hedman



Boost-users list run by williamkempf at hotmail.com, kalb at libertysoft.com, bjorn.karlsson at readsoft.com, gregod at cs.rpi.edu, wekempf at cox.net