Re: [Boost-users] Tokenizer usage, combining escaped_list_ and char_?

From: Nitin Motgi (nitin.motgi_at_[hidden])
Date: 2006-01-24 11:35:17

Jeremy Tudisco wrote:
> I'm attempting to process a scripting language from a file through tokenizer (only recently found out about Spirit, more on that later), and am having difficulties, namely, in processing input like this: 
> "[ ExampleRoutine input;

IMHO Tokenizer should not be used for parsing such syntaxes.

> However, as it stands, escape_list_separator doesn't return the separators, it just acts upon them, so all that fancy operator parsing isn't possible out of the box, requiring breaking things like "switch(" and "input.messageNumber" into separate things. Possible, yes, but extra work.

Yes, it does not return the seperator that just acted upon in tokenizing.

> So, the question. Could my needs be satisfied by defining my own TokenizerFunction, and if so, is there a simpler/more exhaustive reference besides the page?

IMHO it would not be worth doing so. There is no documentation available.

> Or, conversely, is it time to look into Spirit?

I think for these kind of things using spirit would be
elegant. If provides better features for parsing complex
syntaxes.

Thank you,
Nitin Motgi