Re: [boost] Re: Review: Tokenizer

From: Jeremy Siek (jsiek_at_[hidden])
Date: 2001-06-04 09:38:20

On Mon, 4 Jun 2001 jbandela_at_[hidden] wrote:

> So you are basically saying to rename tokenizer_policy.hpp to
> tokenizer_iterator.hpp and put tokenizer_policy in detail?

Yep.

> > The token_iterator class should be deleted. You are attempting to
> create a
> > templated typedef using inheritance, which I'm afraid doesn't
> really work.
> > There are corner cases where bad things happen. Also, the
> > token_iterator_generator already serves the purpose.
>
> Could you give some examples of when this would cause problems? Right
> now I see it as a rather nice convenience class, but I am willing to
> be enlightened.

One of the main problems is that the return type of operator++ is
wrong, it isn't token_iterator. Here's an example:

#include <boost/tokenizer.hpp>
#include <boost/token_functions.hpp>
#include <boost/pending/iterator_tests.hpp>

#if 1
typedef boost::token_iterator<std::string,std::string::const_iterator,
    boost::punct_space_separator<char> > token_iter;
#else
typedef boost::token_iterator_generator<std::string,
    std::string::const_iterator,
    boost::punct_space_separator<char> >::type token_iter;
#endif

void foo(const token_iter&) { }

int main()
{
  using namespace boost;
  using namespace std;
  const string test_string = "This,,is, a.test..";
  const string answer[] = {"This","is","a","test"};
  token_iter begin
    = make_token_iterator<std::string>(test_string.begin(),test_string.end(),
                          boost::punct_space_separator<char>()), end;
  foo(++begin);
}

> Doesn't punct_space_separator provide all the functionality in
> strtok, namely using single characters as delimiters? However, I

I don't think so. strtok() lets the user specify the delimiters, whereas I
thought pucht_space_separator hard-codes in the separators as punctuation
and whitespace.

Cheers,
Jeremy

----------------------------------------------------------------------
 Jeremy Siek                        www: http://www.lsc.nd.edu/~jsiek/
 Ph.D. Candidate, IU B'ton          email: jsiek_at_[hidden]
 Summer Manager, AT&T Research      phone: [phone]
----------------------------------------------------------------------