Hi, Is there any advantage in using both tokenizer and spirit? I'm not a tokenizer expert, but it seems that what you are trying to achieve can be done by spirit alone: parse(first, last, uint_p[append(numbers)], space_p); -- Joel de Guzman joel at boost-consulting.com http://www.boost-consulting.com http://spirit.sf.net Andrey Sverdlichenko <blaze@lists.infosec.ru> wrote:
Friday, July 25, 2003, 3:47:41 AM, you wrote:
Hello.
Is there any "right way" to parse strings, already splitted with tokenizer? I wrote iterator that can work with this splitted strings, but i also need some "skip parser" to recognize token boundaries as whitespaces and can't design one.
Could you please be more specific?
This is a sample code. It parses both numbers in data string as one single number and i need to separate them.
#include <boost/tokenizer.hpp> #include <boost/spirit/core.hpp> #include <list> #include <iostream>
typedef boost::char_separator<std::string::value_type> Separator; typedef boost::tokenizer<Separator> Tokenizer;
class tok_iterator : public std::iterator<std::forward_iterator_tag, const std::string::value_type> { public: explicit tok_iterator(const Tokenizer::iterator &curr) : token(curr), offset(0) {} int operator ==(const tok_iterator &other) const { return (token == other.token && offset == other.offset); } int operator !=(const tok_iterator &other) const { return ! (*this == other); } reference operator *(void) const { return (*token)[offset]; }
tok_iterator &operator ++(void);
private: Tokenizer::iterator token; size_t offset; };
inline tok_iterator & tok_iterator::operator ++(void) { if (++offset >= token->size()) { offset = 0; ++token; } return *this; }
int main(void) { using namespace boost::spirit;
std::string data("55 99"); Separator sep; Tokenizer tok(data, sep); Tokenizer::iterator token = tok.begin(); std::list<u_int> numbers;
parse_info<tok_iterator> info = parse(tok_iterator(token), tok_iterator(tok.end()), uint_p[append(numbers)]);
std::copy(numbers.begin(), numbers.end(), std::ostream_iterator<u_int>(std::cout, "\n"));
return 0; }