Re: [boost] Re: tokenizer profiling results

From: todor_at_[hidden]
Date: 2001-10-20 05:43:34

From: <jbandela_at_[hidden]>

> I have posted a test version of tokenizer to tokenizer_optimize.zip
> in the files section.

Why do you need this BogusIterator<>/is_valid()/set_invalid()
construct?
What's wrong with simply comparing cur() with get_base_end()?

Also, you should not make the assumption that default constructed
token
is invalid. Consider the following fragment from CSV file:
  12, ""
With token_type == std::string this should produce two tokens:
  std::string("12") followed by std::string("")

Todor