Re: Re[4]: [Boost-Users] spirit & tokenizer

From: Joel de Guzman (djowel_at_[hidden])
Date: 2003-07-27 20:02:43

Andrey Sverdlichenko <blaze_at_[hidden]> wrote:
> Saturday, July 26, 2003, 4:08:52 AM, you wrote:
> 
> JdG> Is there any advantage in using both tokenizer and spirit? I'm not a
> JdG> tokenizer expert, but it seems that what you are trying to achieve can
> JdG> be done by spirit alone:
> 
> JdG>     parse(first, last,  uint_p[append(numbers)], space_p);
> 
> It was just a sample. Before this numbers there is command line that
> i prefer to parse with tokenizer (i really have reasons for this) and
> after i split this string there is no way to convert tokenizer::iterator
> to string::iterator.

Hi,

Can we move this discussion over to Spirit's ML?
Spirit-general mailing list
Spirit-general_at_[hidden]
https://lists.sourceforge.net/lists/listinfo/spirit-general

As I said, I am not a tokenizer expert. Anyway, perhaps what you want to do
is to make the tokenizer output your *lexer*. Please check out the C++ lexer
(written by JCAB). It provides a really good example on how to use a lexer
with Spirit. The idea is to pass a *token stream* to Spirit instead of the raw
character stream. For that to work, you'll probably have to do some work with
the tokenizer output to assign individual tokens to the parsed input. For
instance, say we have an input: "123 456 a big brown fox" your token stream
might look like:

    struct token
    {
        char ID; // int, string... etc.
        Iter first; 
        Iter last; 
    };

You'll probably need Spirit to parse the individual lexical tokens from the
tokenizer output. Pseudocode:
    
    vector <token>
    for each tokenizer output
        if parse(tokenizer[i], lex_rule).full
            vector.push_back(token(tokenizer[i]))
        else
            lexical error!!!

Then, pass in your vector of tokens to Spirit. Be sure to have ==, != and <
operators to/from your token::ID so you can write (Example):

    char const INT_TOK= 1;
    char const STR_TOK = 2;

    r = ch_p(INT_TOK) | STR_TOK;
    start_ = *r;

HTH. If you have further questions, let's continue the discussion in Spirit's ML.

Regards,
-- 
Joel de Guzman
joel at boost-consulting.com
http://www.boost-consulting.com
http://spirit.sf.net