Re: [boost] Using lexertl instead of spirit

From: Hartmut Kaiser (hartmut.kaiser_at_[hidden])
Date: 2006-11-20 22:31:55

Joel de Guzman wrote: 

> >> The main selling point of lexertl as far as boost goes is 
> probably speed. 
> >> lexertl lexers are both fast to construct (typically under 100 
> >> milliseconds on a modern machine) and fast to tokenise input (the 
> >> generated state machine uses the flex technique of 
> equivalence classes).
> >>
> >> I personally like the fact that you can dump the DFA as data and 
> >> process it later with any language you like too.
> > 
> > Spirit is a parser, please don't compare apples and oranges.
> > You cannot implement, say, Wave, with just a lexer. I suggest you 
> > don't go the "this-is-better-that-is-better" route. Spirit 
> has its own 
> > lexer too, FYI. It's called SLex. See Slex in 
> http://tinyurl.com/29mcn
> 
> Oh and BTW, if you want to talk about speed, matching the 
> speed of Flex is not good enough. The thing to beat is Re2C! 
> Hartmut and I shall see how lexertl fares soon. Wave has an 
> adaptable front end where you can choose your own lexer. Re2C 
> (http://re2c.org/) is one of them.

Yes, and Slex is the other one. BTW, Slex seems to be very similar to
lexertl (with the exeption of not allowing to optimize the constructed state
machine tables - but this is not a principal issue, merely lack of time). 

Wave is based on a modular (layered) design. The lexer sits on top of the
input (character) stream, producing C++ tokens, exposed via an iterator
interface. The preprocessing component consumes the lexer iterators and is
almost completely independent. The only dependency is that both have to use
the same token type (which is a template parameter to both).

It is very easy to interface a different lexer to the preprocessor. The
cpp_tokens example (libs/wave/samples/cpp_tokens) demonstrates this by using
the Slex based lexer. I'll elaborate on this in another mail during the next
couple of days.

Regards Hartmut