[boost] [ANN] Lexertl based lexer module integrated with Wave

From: Hartmut Kaiser (hartmut.kaiser_at_[hidden])
Date: 2006-12-06 11:17:58

Hi all,

Ben Hanson and I were busy over the last couple of days to build a full C++
lexer based on Ben's Lexertl framework
(http://www.benhanson.net/lexertl.html) which is integrated into Wave (as
proof of concept). Now, as we have accomplished this I'ld like to take the
chance to provide some (rough) performance measurements. As a basis we took
the two existing (functionally equivalent) Wave C++ lexers (based on Re2C
and Slex) and measured the execution times to scan all possible C++ tokens
in a loop 1000 times. Here are the results:

 Re2C:    1.22 [s]
 Slex:    1.89 [s]
 Lexertl: 1.47 [s] (1.35 [s] without minimize_dfa)

These measurements include the construction times of the lexer objects for
each of the scanned C++ tokens.

I'm personally impressed by the performance results of Lexertl. Keep in
mind, Re2C has no DFA generation overhead at runtime and the generated code
is very compiler friendly - essentially a huge switch statement, allowing
for highly optimzed code. So I didn't expect Lexertl to be as fast as Re2C.
But I didn't expect it to be so close to Re2C either.

The bottom line is: I highly encorage Ben to boostify his code since I'm
pretty sure Lexertl would make a very useful and state of the art addition
to Boost.

For those interested to have a peek at the code, it's in the Boost
CVS::HEAD, integrated into the Wave example 'list_includes'. The Lexertl
code itself is available from the website mentioned above. 

Regards Hartmut