Re: Re[2]: [Boost-Users] spirit & tokenizer

From: Joel de Guzman (djowel_at_[hidden])
Date: 2003-07-25 19:08:52

Hi,

Is there any advantage in using both tokenizer and spirit? I'm not a
tokenizer expert, but it seems that what you are trying to achieve can
be done by spirit alone:

    parse(first, last,  uint_p[append(numbers)], space_p);

-- 
Joel de Guzman
joel at boost-consulting.com
http://www.boost-consulting.com
http://spirit.sf.net
Andrey Sverdlichenko <blaze_at_[hidden]> wrote:
> Friday, July 25, 2003, 3:47:41 AM, you wrote:
> 
>>>   Hello.
>>> 
>>>   Is there any "right way" to parse strings, already splitted with
>>>   tokenizer? I wrote iterator that can work with this splitted
>>>   strings, but i also need some "skip parser" to recognize token
>>>   boundaries as whitespaces and can't design one.
> 
>> Could you please be more specific?
> 
> This is a sample code. It parses both numbers in data string as one
> single number and i need to separate them.
> 
> #include <boost/tokenizer.hpp>
> #include <boost/spirit/core.hpp>
> #include <list>
> #include <iostream>
> 
> typedef boost::char_separator<std::string::value_type> Separator;
> typedef boost::tokenizer<Separator> Tokenizer;
> 
> class tok_iterator : public std::iterator<std::forward_iterator_tag,
>                                 const std::string::value_type> {
> public:
>         explicit tok_iterator(const Tokenizer::iterator &curr)
>                 : token(curr), offset(0) {}
>         int operator ==(const tok_iterator &other) const
>                 { return (token == other.token && offset == other.offset); }
>         int operator !=(const tok_iterator &other) const
>                 { return ! (*this == other); }
>         reference operator *(void) const
>                 { return (*token)[offset]; }
> 
>         tok_iterator &operator ++(void);
> 
> private:
>         Tokenizer::iterator token;
>         size_t offset;
> };
> 
> inline
> tok_iterator &
> tok_iterator::operator ++(void) {
>         if (++offset >= token->size()) {
>                 offset = 0;
>                 ++token;
>         }
>         return *this;
> }
> 
> int
> main(void) {
>         using namespace boost::spirit;
> 
>         std::string data("55 99");
>         Separator sep;
>         Tokenizer tok(data, sep);
>         Tokenizer::iterator token = tok.begin();
>         std::list<u_int> numbers;
> 
>         parse_info<tok_iterator> info = parse(tok_iterator(token),
>                                 tok_iterator(tok.end()),
>                                 uint_p[append(numbers)]);
> 
>         std::copy(numbers.begin(), numbers.end(),
>                         std::ostream_iterator<u_int>(std::cout, "\n"));
> 
>         return 0;
> }