[boost] Re: Interest in a fast lexical analyser compatible with Spirit

From: Joel de Guzman (joel_at_[hidden])
Date: 2004-12-28 07:09:16

Dave Handley wrote:

> Joel de Guzman wrote:
> 
>> Some more things to note:
>> * why use lexeme_d on the lexer stage?
>> * the comment rule can be rewritten as:
>>   '#' >> *(anychar_p - eol_p) >> eol_p;
>> * stringLiteral can be rewriten similarly.
>> * word can be rewritten using chsets.
> 
> 
> I used lexeme_d because the VRML grammar uses whitespace to separate 
> ints and floats.  Without the lexeme_d, ints could merge into one.

I smell something wrong here. Lexers work at the character level
and process white space appropriately. The added skip-parser will
be another source of slowdown. White-space skipping should be in
the lexer, not outside it.

> Joel de Guzman wrote:
> 
>> Sure, no problem. But watch out, Spirit-2 will have a speed improvement
>> in the 10X-20X range over classic Spirit ;-) I already have some test
>> cases, and this does not touch on predictive parsing and lexing.
> 
> 
> Joel,  I would be really keen to try running my VRML test case through 
> Spirit-2.  Given that I've now got the Spirit-1 parser running at about 
> 6-7 seconds, if you get a 10x speed improvement on this file, then it 
> could make the case for separate lexing much harder to support!  I may 
> also need a bigger file to start doing test cases on - millisecond test 
> timings don't make for accuracy in my view :-)

See the smiley ;-) ... I was just teasing :-) I do have some rather
informal test cases though, on a toy parser. Seriously, well, it's
rather early to say anything really conclusive. I'm not yet sure how it
would turn out in real world cases outside the toy room. I do intend
to do lots of benchmarks and optimization. That's an area
that's largely left for later. The initial thrust was not on speed,
but on the interface and usability. We've certainly learned a lot
from the start when Spirit debuted. Now it's time to rock and roll!

> Joel de Guzman wrote:
> 
>> Anyway, as I said, I'm very interested with your lexer.
>> Would you be interested in a collaboration?
> 
> 
> Yes, we would be very interested in a collaboration.

Very cool! I'll keep in touch as things progress in the Spirit-2
world. Lexing + predictive parsing: that's where I'm heading.

Regards,
-- 
Joel de Guzman
http://www.boost-consulting.com
http://spirit.sf.net