Boost-Commit :

Date view	Thread view	Subject view	Author view

From: hartmut.kaiser_at_[hidden]
Date: 2008-07-14 21:20:38

Next message: widman_at_[hidden]: "[Boost-commit] svn:boost r47429 - sandbox/committee/concepts/issues"
Previous message: dgregor_at_[hidden]: "[Boost-commit] svn:boost r47427 - branches/CMake/release/libs/accumulators"

Author: hkaiser
Date: 2008-07-14 21:20:34 EDT (Mon, 14 Jul 2008)
New Revision: 47428
URL: http://svn.boost.org/trac/boost/changeset/47428

Log:
Spirit: more work on docs
Added:
   trunk/libs/spirit/doc/html/spirit/lex/
   trunk/libs/spirit/doc/html/spirit/lex.html (contents, props changed)
   trunk/libs/spirit/doc/html/spirit/lex/abstracts/
   trunk/libs/spirit/doc/html/spirit/lex/abstracts.html (contents, props changed)
   trunk/libs/spirit/doc/html/spirit/lex/abstracts/lexer_attributes.html (contents, props changed)
   trunk/libs/spirit/doc/html/spirit/lex/abstracts/lexer_parsing.html (contents, props changed)
   trunk/libs/spirit/doc/html/spirit/lex/abstracts/lexer_semantic_actions.html (contents, props changed)
   trunk/libs/spirit/doc/html/spirit/lex/abstracts/lexer_states.html (contents, props changed)
   trunk/libs/spirit/doc/html/spirit/lex/abstracts/lexer_static_model.html (contents, props changed)
   trunk/libs/spirit/doc/html/spirit/lex/abstracts/lexer_tokenizing.html (contents, props changed)
   trunk/libs/spirit/doc/html/spirit/lex/quick_reference.html (contents, props changed)
   trunk/libs/spirit/doc/html/spirit/lex/reference/
   trunk/libs/spirit/doc/html/spirit/lex/reference.html (contents, props changed)
   trunk/libs/spirit/doc/html/spirit/lex/reference/lexer_class.html (contents, props changed)
   trunk/libs/spirit/doc/html/spirit/lex/reference/token_class.html (contents, props changed)
   trunk/libs/spirit/doc/html/spirit/lex/reference/tokendef_class.html (contents, props changed)
   trunk/libs/spirit/doc/html/spirit/lex/reference/tokenset_class.html (contents, props changed)
   trunk/libs/spirit/doc/html/spirit/lex/tutorials/
   trunk/libs/spirit/doc/html/spirit/lex/tutorials.html (contents, props changed)
   trunk/libs/spirit/doc/html/spirit/lex/tutorials/lexer_quickstart1.html (contents, props changed)
   trunk/libs/spirit/doc/html/spirit/lex/tutorials/lexer_quickstart2.html (contents, props changed)
   trunk/libs/spirit/doc/html/spirit/lex/tutorials/lexer_quickstart3.html (contents, props changed)
Removed:
   trunk/libs/spirit/doc/html/spirit/__lex__/
   trunk/libs/spirit/doc/html/spirit/__lex__.html
Text files modified:
   trunk/libs/spirit/doc/html/index.html | 73 +++++++++++++++++++--------------------
   trunk/libs/spirit/doc/html/spirit/acknowledgments.html | 2
   trunk/libs/spirit/doc/html/spirit/faq.html | 6 +-
   trunk/libs/spirit/doc/html/spirit/introduction.html | 8 ++--
   trunk/libs/spirit/doc/html/spirit/qi_and_karma.html | 67 +++++++++++++++++++-----------------
   trunk/libs/spirit/doc/html/spirit/qi_and_karma/abstracts/mini_xml___error_handling.html | 6 +-
   trunk/libs/spirit/doc/html/spirit/qi_and_karma/abstracts/parsing.html | 10 ++--
   trunk/libs/spirit/doc/html/spirit/qi_and_karma/abstracts/parsing_and_generating.html | 10 ++--
   trunk/libs/spirit/doc/html/spirit/qi_and_karma/quick_reference.html | 2
   trunk/libs/spirit/doc/html/spirit/qi_and_karma/reference.html | 10 ++--
   trunk/libs/spirit/doc/html/spirit/qi_and_karma/reference/debug.html | 6 +-
   trunk/libs/spirit/doc/html/spirit/what_s_new.html | 2
   trunk/libs/spirit/doc/lex.qbk | 12 +++---
   trunk/libs/spirit/doc/lex/introduction.qbk | 21 ++++++++++-
   trunk/libs/spirit/doc/lex/lexer_attributes.qbk | 2
   trunk/libs/spirit/doc/lex/lexer_primitives.qbk | 3 +
   trunk/libs/spirit/doc/lex/lexer_quickstart1.qbk | 2
   trunk/libs/spirit/doc/lex/lexer_quickstart2.qbk | 2
   trunk/libs/spirit/doc/lex/lexer_quickstart3.qbk | 2
   trunk/libs/spirit/doc/lex/lexer_semantic_actions.qbk | 2
   trunk/libs/spirit/doc/lex/lexer_states.qbk | 2
   trunk/libs/spirit/doc/lex/lexer_static_model.qbk | 2
   trunk/libs/spirit/doc/lex/lexer_tutorials.qbk | 2
   trunk/libs/spirit/doc/lex/parsing_using_a_lexer.qbk | 2
   trunk/libs/spirit/doc/lex/token_definition.qbk | 2
   trunk/libs/spirit/doc/lex/tokenizing.qbk | 2
   trunk/libs/spirit/doc/lex/tokens_values.qbk | 2
   trunk/libs/spirit/doc/spirit2.qbk | 30 ++++++++--------
   28 files changed, 156 insertions(+), 136 deletions(-)

Modified: trunk/libs/spirit/doc/html/index.html
==============================================================================
--- trunk/libs/spirit/doc/html/index.html (original)
+++ trunk/libs/spirit/doc/html/index.html 2008-07-14 21:20:34 EDT (Mon, 14 Jul 2008)
@@ -34,7 +34,7 @@
<div>Copyright © 2001-2008 Joel de Guzman,
 Hartmut Kaiser</div>
<div><div class="legalnotice">
-<a name="id455998"></a>
+<a name="id638057"></a>
 Distributed under the Boost Software License, Version 1.0. (See accompanying
 file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
 
@@ -113,44 +113,43 @@
<dt>Debug</dt>
</dl></dd>
</dl></dd>
-<dt>Spirit.Lex</dt>
+<dt> Spirit.Lex</dt>
<dd><dl>
-<dt><a href="spirit/__lex__.html#spirit.__lex__.introduction_to___lex__">Introduction
- to Spirit.Lex</a></dt>
-<dt><a href="spirit/__lex__/__lex___tutorials.html">Spirit.Lex
+<dt> Introduction to Spirit.Lex</dt>
+<dt><a href="spirit/lex/tutorials.html"> Spirit.Lex
 Tutorials</a></dt>
<dd><dl>
-<dt><a href="spirit/__lex__/__lex___tutorials.html#spirit.__lex__.__lex___tutorials.__lex___tutorials_overview">Spirit.Lex
+<dt><a href="spirit/lex/tutorials.html#spirit.lex.tutorials.lexer_tutorials"> Spirit.Lex
 Tutorials Overview</a></dt>
-<dt><a href="spirit/__lex__/__lex___tutorials/quickstart_1___a_word_counter_using___lex__.html">Quickstart
+<dt><a href="spirit/lex/tutorials/lexer_quickstart1.html"> Quickstart
 1 - A word counter using Spirit.Lex</a></dt>
-<dt><a href="spirit/__lex__/__lex___tutorials/quickstart_2___a_better_word_counter_using___lex__.html">Quickstart
+<dt><a href="spirit/lex/tutorials/lexer_quickstart2.html"> Quickstart
 2 - A better word counter using Spirit.Lex</a></dt>
-<dt><a href="spirit/__lex__/__lex___tutorials/quickstart_3___counting_words_using_a_parser.html">Quickstart
+<dt><a href="spirit/lex/tutorials/lexer_quickstart3.html"> Quickstart
 3 - Counting Words Using a Parser</a></dt>
</dl></dd>
-<dt>Abstracts</dt>
+<dt> Abstracts</dt>
<dd><dl>
-<dt>Lexer Primitives</dt>
-<dt><a href="spirit/__lex__/abstracts/tokenizing_input_data.html">Tokenizing
+<dt>Lexer Primitives</dt>
+<dt><a href="spirit/lex/abstracts/lexer_tokenizing.html"> Tokenizing
 Input Data</a></dt>
-<dt><a href="spirit/__lex__/abstracts/lexer_semantic_actions.html">Lexer
+<dt><a href="spirit/lex/abstracts/lexer_semantic_actions.html"> Lexer
 Semantic Actions</a></dt>
-<dt><a href="spirit/__lex__/abstracts/the__static__lexer_model.html">The
- Static Lexer Model</a></dt>
-<dt><a href="spirit/__lex__/abstracts/parsing_using_a_lexer.html">Parsing
- using a Lexer</a></dt>
-<dt>Lexer Attributes</dt>
-<dt>Lexer States</dt>
+<dt><a href="spirit/lex/abstracts/lexer_static_model.html"> The Static
+ Lexer Model</a></dt>
+<dt><a href="spirit/lex/abstracts/lexer_parsing.html"> Parsing using
+ a Lexer</a></dt>
+<dt> Lexer Attributes</dt>
+<dt> Lexer States</dt>
</dl></dd>
-<dt>Quick Reference</dt>
-<dt>Reference</dt>
+<dt> Quick Reference</dt>
+<dt> Reference</dt>
<dd><dl>
-<dt>Concepts</dt>
-<dt>Lexer Class</dt>
-<dt>Token Class</dt>
-<dt>TokenDef Class</dt>
-<dt>TokenSet Class</dt>
+<dt> Concepts</dt>
+<dt>Lexer Class</dt>
+<dt>Token Class</dt>
+<dt>TokenDef Class</dt>
+<dt>TokenSet Class</dt>
</dl></dd>
</dl></dd>
<dt>FAQ</dt>
@@ -185,11 +184,11 @@
 
</blockquote></div>
<a name="spirit.preface.history"></a><h3>
-<a name="id385825"></a>
+<a name="id638111"></a>
 <a href="index.html#spirit.preface.history">History</a>
 </h3>
<a name="spirit.preface._emphasis_80s__emphasis_"></a><h3>
-<a name="id385846"></a>
+<a name="id638124"></a>
 <a href="index.html#spirit.preface._emphasis_80s__emphasis_">80s</a>
 </h3>

@@ -203,7 +202,7 @@
 recursive-descent parser. This worked very well.
 
<a name="spirit.preface._emphasis_90s__emphasis_"></a><h3>
-<a name="id386884"></a>
+<a name="id638142"></a>
 <a href="index.html#spirit.preface._emphasis_90s__emphasis_">90s</a>
 </h3>

@@ -248,7 +247,7 @@
 here: pre-Spirit.
 
<a name="spirit.preface._emphasis_2001_to_2006__emphasis_"></a><h3>
-<a name="id387194"></a>
+<a name="id638185"></a>
 <a href="index.html#spirit.preface._emphasis_2001_to_2006__emphasis_">2001
 to 2006</a>
 </h3>
@@ -278,7 +277,7 @@
 for design and implementation quality, robustness, and reusability.
 
<a name="spirit.preface._emphasis_2007__emphasis_"></a><h3>
-<a name="id387270"></a>
+<a name="id638220"></a>
 <a href="index.html#spirit.preface._emphasis_2007__emphasis_">2007</a>
 </h3>

@@ -308,7 +307,7 @@
 Templates</a> library called -Boost.Proto-.
 
<a name="spirit.preface.new_ideas__spirit_v2"></a><h3>
-<a name="id387389"></a>
+<a name="id638286"></a>
 <a href="index.html#spirit.preface.new_ideas__spirit_v2">New Ideas: Spirit V2</a>
 </h3>

@@ -346,12 +345,12 @@
 same underlying component library as for the parser and generator libraries.
 
<a name="spirit.preface.how_to_use_this_manual"></a><h3>
-<a name="id387492"></a>
+<a name="id638337"></a>
 <a href="index.html#spirit.preface.how_to_use_this_manual">How to use this manual</a>
 </h3>

 Each major section (there are two: <a href="spirit/qi_and_karma.html" title="Qi and Karma">Qi and
- Karma</a>, and Lex) is roughly divided
+ Karma</a>, and Lex) is roughly divided
 into 3 parts:
 
<div class="orderedlist"><ol type="1">
@@ -377,7 +376,7 @@
 icons precede some text to indicate:
 
<div class="table">
-<a name="id387578"></a>Table 1. Icons
+<a name="id638385"></a>Table 1. Icons
<table class="table" summary="Icons">
<colgroup>
<col>
@@ -498,7 +497,7 @@
 Tools</a>.
 
<a name="spirit.preface.support"></a><h3>
-<a name="id385560"></a>
+<a name="id638626"></a>
 <a href="index.html#spirit.preface.support">Support</a>
 </h3>

@@ -514,7 +513,7 @@
</div>
</div>
<table xmlns:rev="http://www.cs.rpi.edu/~gregod/boost/tools/doc/revision" width="100%"><tr>
-<td align="left">Last revised: July 13, 2008 at 06:03:25 GMT</td>
+<td align="left">Last revised: July 15, 2008 at 01:18:38 GMT</td>
<td align="right"><div class="copyright-footer"></div></td>
</tr></table>
<hr>

Deleted: trunk/libs/spirit/doc/html/spirit/__lex__.html
==============================================================================
--- trunk/libs/spirit/doc/html/spirit/__lex__.html 2008-07-14 21:20:34 EDT (Mon, 14 Jul 2008)
+++ (empty file)
@@ -1,266 +0,0 @@
-<html>
-<head>
-<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
-<title>Spirit.Lex</title>
-<link rel="stylesheet" href="../../../../../doc/html/boostbook.css" type="text/css">
-<meta name="generator" content="DocBook XSL Stylesheets V1.66.1">
-<link rel="start" href="../index.html" title="Spirit 2.0">
-<link rel="up" href="../index.html" title="Spirit 2.0">
-<link rel="prev" href="qi_and_karma/reference/debug.html" title="Debug">
-<link rel="next" href="__lex__/__lex___tutorials.html" title="Spirit.Lex
- Tutorials">
-</head>
-<body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF">
-<table cellpadding="2" width="100%"><tr>
-<td valign="top"><img alt="Boost C++ Libraries" width="277" height="86" src="../../../../../boost.png"></td>
-<td align="center">Home</td>
-<td align="center">Libraries</td>
-<td align="center">People</td>
-<td align="center">FAQ</td>
-<td align="center">More</td>
-</tr></table>
-<hr>
-<div class="spirit-nav">
-<a accesskey="p" href="qi_and_karma/reference/debug.html"><img src="../../../../../doc/html/images/prev.png" alt="Prev"></a><a accesskey="u" href="../index.html"><img src="../../../../../doc/html/images/up.png" alt="Up"></a><a accesskey="h" href="../index.html"><img src="../../../../../doc/html/images/home.png" alt="Home"></a><a accesskey="n" href="__lex__/__lex___tutorials.html"><img src="../../../../../doc/html/images/next.png" alt="Next"></a>
-</div>
-<div class="section" lang="en">
-<div class="titlepage"><div><div><h2 class="title" style="clear: both">
-<a name="spirit.__lex__"></a>Spirit.Lex</h2></div></div></div>
-<div class="toc"><dl>
-<dt><a href="__lex__.html#spirit.__lex__.introduction_to___lex__">Introduction
- to Spirit.Lex</a></dt>
-<dt><a href="__lex__/__lex___tutorials.html">Spirit.Lex
- Tutorials</a></dt>
-<dd><dl>
-<dt><a href="__lex__/__lex___tutorials.html#spirit.__lex__.__lex___tutorials.__lex___tutorials_overview">Spirit.Lex
- Tutorials Overview</a></dt>
-<dt><a href="__lex__/__lex___tutorials/quickstart_1___a_word_counter_using___lex__.html">Quickstart
- 1 - A word counter using Spirit.Lex</a></dt>
-<dt><a href="__lex__/__lex___tutorials/quickstart_2___a_better_word_counter_using___lex__.html">Quickstart
- 2 - A better word counter using Spirit.Lex</a></dt>
-<dt><a href="__lex__/__lex___tutorials/quickstart_3___counting_words_using_a_parser.html">Quickstart
- 3 - Counting Words Using a Parser</a></dt>
-</dl></dd>
-<dt>Abstracts</dt>
-<dd><dl>
-<dt>Lexer Primitives</dt>
-<dt><a href="__lex__/abstracts/tokenizing_input_data.html">Tokenizing
- Input Data</a></dt>
-<dt><a href="__lex__/abstracts/lexer_semantic_actions.html">Lexer
- Semantic Actions</a></dt>
-<dt><a href="__lex__/abstracts/the__static__lexer_model.html">The
- Static Lexer Model</a></dt>
-<dt><a href="__lex__/abstracts/parsing_using_a_lexer.html">Parsing
- using a Lexer</a></dt>
-<dt>Lexer Attributes</dt>
-<dt>Lexer States</dt>
-</dl></dd>
-<dt>Quick Reference</dt>
-<dt>Reference</dt>
-<dd><dl>
-<dt>Concepts</dt>
-<dt>Lexer Class</dt>
-<dt>Token Class</dt>
-<dt>TokenDef Class</dt>
-<dt>TokenSet Class</dt>
-</dl></dd>
-</dl></div>
-<div class="section" lang="en">
-<div class="titlepage"><div><div><h3 class="title">
-<a name="spirit.__lex__.introduction_to___lex__"></a><a href="__lex__.html#spirit.__lex__.introduction_to___lex__" title="Introduction
- to Spirit.Lex">Introduction
- to Spirit.Lex</a></h3></div></div></div>
-
- Lexical scanning is the process of analyzing the stream of input characters
- and separating it into strings called tokens, separated by whitespace. Most
- compiler texts start here, and devote several chapters to discussing various
- ways to build scanners. Spirit.Lex is a library built
- to take care of the complexities of creating a lexer for your grammar (in
- this documentation we will use the terms 'lexical analyzer', 'lexer' and
- 'scanner' interchangably). All it needs to create a lexer is to know the
- set of patterns describing the different tokens you want to recognize in
- the input. To make this a bit more formal, here are some definitions:
- 
-<div class="itemizedlist"><ul type="disc">
-<li>
- A token is a sequence of consecutive characters having a collective meaning.
- Tokens may have attributes specific to the token type, carrying additional
- information about the matched character sequence.
- </li>
-<li>
- A pattern is a rule expressed as a regular expression and describing how
- a particular token can be formed. For example, <tt class="literal">[A-Za-z][A-Za-z_0-9]*</tt>
- is a pattern for a rule matching C++ identifiers.
- </li>
-<li>
- Characters between tokens are called whitespace; these include spaces,
- tabs, newlines, and formfeeds. Many people also count comments as whitespace,
- though since some tools such as lint look at comments, this conflation
- is not perfect.
- </li>
-</ul></div>
-<a name="spirit.__lex__.introduction_to___lex__.why_using_a_separate_lexer"></a><h4>
-<a name="id487196"></a>
- <a href="__lex__.html#spirit.__lex__.introduction_to___lex__.why_using_a_separate_lexer">Why
- Using a Separate Lexer</a>
- </h4>
-
- Typically, lexical scanning is done in a separate module from the parser,
- feeding the parser with a stream of input tokens only. Now, theoretically
- it is not necessary to do this separation. In the end there is only one set
- of syntactical rules defining the language, so in theory we could write the
- whole parser in one module. In fact, Spirit.Qi allows
- to write parsers without using a lexer, parsing the input character stream
- directly, and for the most part this is the way Spirit
- has been used since its invention.
- 
-
- However, the separation has both practical and theoretical bases and proves
- to be very useful in practical applications. In 1956, Noam Chomsky defined
- the "Chomsky Hierarchy" of grammars:
- 
-<div class="itemizedlist"><ul type="disc">
-<li>
- Type 0: Unrestricted grammars (e.g., natural languages)
- </li>
-<li>
- Type 1: Context-Sensitive grammars
- </li>
-<li>
- Type 2: Context-Free grammars
- </li>
-<li>
- Type 3: Regular grammars
- </li>
-</ul></div>
-
- The complexity of these grammars increases from regular grammars being the
- simplest to unrestricted grammars being the most complex. Similarily, the
- complexity of the recognizers for these grammars increases. Although, a few
- features of some programming languages (such as C++) are Type 1, fortunately
- for the most part programming languages can be described using only the Types
- 3 and 2. The neat part about these two types is that they are well known
- and the ways to parse them are well understood. It has been shown that any
- regular grammar can be parsed using a state machine (finite automaton). Similarly,
- context-free grammars can always be parsed using a push-down automaton (essentially
- a state machine augmented by a stack).
- 
-
- In real programming languages and practical grammars the parts that can be
- handled as regular expressions tend to be the lower-level parts, such as
- the definition of an identifier or of an integer value:
- 
-<pre class="programlisting">
-letter := [a-zA-Z]
-digit := [0-9]
-
-identifier := letter [ letter | digit ]*
-integer := digit*
-</pre>
-
- Higher level parts of practical grammars tend to be more complex and can't
- be implemented using plain regular expressions anymore. We need to store
- information on the built-in hardware stack while recursing the grammar hierarchy,
- and that in fact this is the preferred approach used for top-down parsing.
- Since it takes a different kind of abstract machine to parse the two types
- of grammars, it proved to be efficient to separate the lexical scanner into
- a separate module which is built around the idea of a state machine. The
- goal here is to use the simplest parsing technique needed for the job.
- 
-
- Another, more practical reason for separating the scanner from the parser
- is the need for backtracking during parsing. The input data is a stream of
- characters, which is often thought to be processed left to right without
- any backtracking. Unfortunately, in practice most of the time that isn't
- possible. Almost every language has certain keywords such as IF, FOR, and
- WHILE. The decision if a certain character sequence actually comprises a
- keyword or just an identifier often can be made only after seeing the first
- delimiter after it. This already is a limited form of
- backtracking, since we need to store the string long enough to be able to
- make the decision. The same is true for more coarse grained language features
- such as nested IF/ELSE statements, where the decision about to which IF belongs
- the last ELSE statement can be made only after seeing the whole construct.
- 
-
- So the structure of a conventional compiler often involves splitting up the
- functions of the lower-level and higher-level parsing. The lexical scanner
- deals with things at the character level, collecting characters into strings,
- converting character sequence into different representations as integers,
- etc., and passing them along to the parser proper as indivisible tokens.
- It's also considered normal to let the scanner do additional jobs, such as
- identifying keywords, storing identifiers in tables, etc.
- 
-
- Now, Spirit follows this
- structure, where Spirit.Lex can be used to implement
- state machine based recognizers, while Spirit.Qi can
- be used to build recognizers for context free grammars. Since both modules
- are seemlessly integrated with each other and with the C++ target language
- it is even possible to use the provided functionality to build more complex
- grammar recognizers.
- 
-<a name="spirit.__lex__.introduction_to___lex__.advantages_of_using__emphasis_spirit_lex__emphasis_"></a><h4>
-<a name="id487571"></a>
- <a href="__lex__.html#spirit.__lex__.introduction_to___lex__.advantages_of_using__emphasis_spirit_lex__emphasis_">Advantages
- of using Spirit.Lex</a>
- </h4>
-
- The advantage of using Spirit.Lex to create the lexical
- analyzer over using more traditional tools such as Flex
- is its carefully crafted integration with the Spirit
- library and the C++ host language. You don't need any external tools to generate
- the code, your lexer will be perfectly integrated with the rest of your program,
- making it possible to freely access any context information and data structure.
- Since the C++ compiler sees all the code it will generate optimal code nomatter
- what configuration options have been chosen by the user. Spirit.Lex
- gives you all the features you could get from a similar Flex
- program without the need to leave C++ as a host language:
- 
-<div class="itemizedlist"><ul type="disc">
-<li>
- the definition of tokens is done using regular expressions (patterns)
- </li>
-<li>
- the token definitions can refer to special substitution string (pattern
- macros) simplifying pattern definitions
- </li>
-<li>
- the generated lexical scanner may have multiple start states
- </li>
-<li>
- it is possible to attach code to any of the token definitions; this code
- gets executed whenever the corresponding token pattern has been matched
- </li>
-</ul></div>
-
- Even if it is possible to use Spirit.Lex to generate
- C++ code representing the lexical analyzer (we will refer to that as the
- static model, described in more detail in the section
- <a href="__lex__/abstracts/the__static__lexer_model.html" title="The
- Static Lexer Model">The Static
- Model</a>) - a model very similar to the way Flex
- operates - we will mainly focus on the opposite, the dynamic
- model. You can directly integrate the token definitions into your C++ program,
- building the lexical analyzer dynamicly at runtime. The dynamic model is
- something not supported by Flex
- or other lexical scanner generators (such as re2c,
- Ragel, etc.).
- But it is very flexible and allows to speed up the development of your application.
- 
-</div>
-</div>
-<table xmlns:rev="http://www.cs.rpi.edu/~gregod/boost/tools/doc/revision" width="100%"><tr>
-<td align="left"></td>
-<td align="right"><div class="copyright-footer">Copyright © 2001-2008 Joel de Guzman,
- Hartmut Kaiser
- Distributed under the Boost Software License, Version 1.0. (See accompanying
- file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
- 
-</div></td>
-</tr></table>
-<hr>
-<div class="spirit-nav">
-<a accesskey="p" href="qi_and_karma/reference/debug.html"><img src="../../../../../doc/html/images/prev.png" alt="Prev"></a><a accesskey="u" href="../index.html"><img src="../../../../../doc/html/images/up.png" alt="Up"></a><a accesskey="h" href="../index.html"><img src="../../../../../doc/html/images/home.png" alt="Home"></a><a accesskey="n" href="__lex__/__lex___tutorials.html"><img src="../../../../../doc/html/images/next.png" alt="Next"></a>
-</div>
-</body>
-</html>

Modified: trunk/libs/spirit/doc/html/spirit/acknowledgments.html
==============================================================================
--- trunk/libs/spirit/doc/html/spirit/acknowledgments.html (original)
+++ trunk/libs/spirit/doc/html/spirit/acknowledgments.html 2008-07-14 21:20:34 EDT (Mon, 14 Jul 2008)
@@ -46,7 +46,7 @@
 FIXME: Add more people
 
<a name="spirit.acknowledgments.acknowledgements_from_the_spirit_v1__emphasis_classic__emphasis__documentation"></a><h3>
-<a name="id505783"></a>
+<a name="id687287"></a>
 <a href="acknowledgments.html#spirit.acknowledgments.acknowledgements_from_the_spirit_v1__emphasis_classic__emphasis__documentation">Acknowledgements
 from the Spirit V1 classic Documentation</a>
 </h3>

Modified: trunk/libs/spirit/doc/html/spirit/faq.html
==============================================================================
--- trunk/libs/spirit/doc/html/spirit/faq.html (original)
+++ trunk/libs/spirit/doc/html/spirit/faq.html 2008-07-14 21:20:34 EDT (Mon, 14 Jul 2008)
@@ -6,7 +6,7 @@
<meta name="generator" content="DocBook XSL Stylesheets V1.66.1">
<link rel="start" href="../index.html" title="Spirit 2.0">
<link rel="up" href="../index.html" title="Spirit 2.0">
-<link rel="prev" href="__lex__/reference/tokenset_class.html" title="TokenSet Class">
+<link rel="prev" href="lex/reference/tokenset_class.html" title="TokenSet Class">
<link rel="next" href="notes.html" title="Notes">
</head>
<body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF">
@@ -20,7 +20,7 @@
</tr></table>
<hr>
<div class="spirit-nav">
-<a accesskey="p" href="__lex__/reference/tokenset_class.html"><img src="../../../../../doc/html/images/prev.png" alt="Prev"></a><a accesskey="u" href="../index.html"><img src="../../../../../doc/html/images/up.png" alt="Up"></a><a accesskey="h" href="../index.html"><img src="../../../../../doc/html/images/home.png" alt="Home"></a><a accesskey="n" href="notes.html"><img src="../../../../../doc/html/images/next.png" alt="Next"></a>
+<a accesskey="p" href="lex/reference/tokenset_class.html"><img src="../../../../../doc/html/images/prev.png" alt="Prev"></a><a accesskey="u" href="../index.html"><img src="../../../../../doc/html/images/up.png" alt="Up"></a><a accesskey="h" href="../index.html"><img src="../../../../../doc/html/images/home.png" alt="Home"></a><a accesskey="n" href="notes.html"><img src="../../../../../doc/html/images/next.png" alt="Next"></a>
</div>
<div class="section" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both">
<a name="spirit.faq"></a>FAQ</h2></div></div></div></div>
@@ -35,7 +35,7 @@
</tr></table>
<hr>
<div class="spirit-nav">
-<a accesskey="p" href="__lex__/reference/tokenset_class.html"><img src="../../../../../doc/html/images/prev.png" alt="Prev"></a><a accesskey="u" href="../index.html"><img src="../../../../../doc/html/images/up.png" alt="Up"></a><a accesskey="h" href="../index.html"><img src="../../../../../doc/html/images/home.png" alt="Home"></a><a accesskey="n" href="notes.html"><img src="../../../../../doc/html/images/next.png" alt="Next"></a>
+<a accesskey="p" href="lex/reference/tokenset_class.html"><img src="../../../../../doc/html/images/prev.png" alt="Prev"></a><a accesskey="u" href="../index.html"><img src="../../../../../doc/html/images/up.png" alt="Up"></a><a accesskey="h" href="../index.html"><img src="../../../../../doc/html/images/home.png" alt="Home"></a><a accesskey="n" href="notes.html"><img src="../../../../../doc/html/images/next.png" alt="Next"></a>
</div>
</body>
</html>

Modified: trunk/libs/spirit/doc/html/spirit/introduction.html
==============================================================================
--- trunk/libs/spirit/doc/html/spirit/introduction.html (original)
+++ trunk/libs/spirit/doc/html/spirit/introduction.html 2008-07-14 21:20:34 EDT (Mon, 14 Jul 2008)
@@ -133,7 +133,7 @@

 
<a name="spirit.introduction.a_quick_overview_about_parsing_with__emphasis_spirit_qi__emphasis_"></a><h3>
-<a name="id456629"></a>
+<a name="id639083"></a>
 <a href="introduction.html#spirit.introduction.a_quick_overview_about_parsing_with__emphasis_spirit_qi__emphasis_">A
 quick overview about Parsing with Spirit.Qi</a>
 </h3>
@@ -228,7 +228,7 @@
 <tt class="computeroutput">;</tt>.
 
<a name="spirit.introduction.a_quick_overview_about_output_generation_with__emphasis_spirit_karma__emphasis_"></a><h3>
-<a name="id457889"></a>
+<a name="id641932"></a>
 <a href="introduction.html#spirit.introduction.a_quick_overview_about_output_generation_with__emphasis_spirit_karma__emphasis_">A
 quick overview about Output Generation with Spirit.Karma</a>
 </h3>
@@ -263,7 +263,7 @@
 of the same <tt class="computeroutput">std::vector<int></tt>:
 
<div class="table">
-<a name="id458364"></a>Table 2. Different output formats for `std::vector<int>`
+<a name="id642275"></a>Table 2. Different output formats for `std::vector<int>`
<table class="table" summary="Different output formats for `std::vector<int>`">
<colgroup>
<col>
@@ -378,7 +378,7 @@
 next table gives some related examples.
 
<div class="table">
-<a name="id458834"></a>Table 3. Different data types usable with the output format
+<a name="id642648"></a>Table 3. Different data types usable with the output format
 `(*int_ << eol)`
<table class="table" summary="Different data types usable with the output format
 `(*int_ << eol)`">

Added: trunk/libs/spirit/doc/html/spirit/lex.html
==============================================================================
--- (empty file)
+++ trunk/libs/spirit/doc/html/spirit/lex.html 2008-07-14 21:20:34 EDT (Mon, 14 Jul 2008)
@@ -0,0 +1,294 @@
+<html>
+<head>
+<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
+<title> Spirit.Lex</title>
+<link rel="stylesheet" href="../../../../../doc/html/boostbook.css" type="text/css">
+<meta name="generator" content="DocBook XSL Stylesheets V1.66.1">
+<link rel="start" href="../index.html" title="Spirit 2.0">
+<link rel="up" href="../index.html" title="Spirit 2.0">
+<link rel="prev" href="qi_and_karma/reference/debug.html" title="Debug">
+<link rel="next" href="lex/tutorials.html" title=" Spirit.Lex
+ Tutorials">
+</head>
+<body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF">
+<table cellpadding="2" width="100%"><tr>
+<td valign="top"><img alt="Boost C++ Libraries" width="277" height="86" src="../../../../../boost.png"></td>
+<td align="center">Home</td>
+<td align="center">Libraries</td>
+<td align="center">People</td>
+<td align="center">FAQ</td>
+<td align="center">More</td>
+</tr></table>
+<hr>
+<div class="spirit-nav">
+<a accesskey="p" href="qi_and_karma/reference/debug.html"><img src="../../../../../doc/html/images/prev.png" alt="Prev"></a><a accesskey="u" href="../index.html"><img src="../../../../../doc/html/images/up.png" alt="Up"></a><a accesskey="h" href="../index.html"><img src="../../../../../doc/html/images/home.png" alt="Home"></a><a accesskey="n" href="lex/tutorials.html"><img src="../../../../../doc/html/images/next.png" alt="Next"></a>
+</div>
+<div class="section" lang="en">
+<div class="titlepage"><div><div><h2 class="title" style="clear: both">
+<a name="spirit.lex"></a> Spirit.Lex</h2></div></div></div>
+<div class="toc"><dl>
+<dt> Introduction to Spirit.Lex</dt>
+<dt><a href="lex/tutorials.html"> Spirit.Lex
+ Tutorials</a></dt>
+<dd><dl>
+<dt><a href="lex/tutorials.html#spirit.lex.tutorials.lexer_tutorials"> Spirit.Lex
+ Tutorials Overview</a></dt>
+<dt><a href="lex/tutorials/lexer_quickstart1.html"> Quickstart
+ 1 - A word counter using Spirit.Lex</a></dt>
+<dt><a href="lex/tutorials/lexer_quickstart2.html"> Quickstart
+ 2 - A better word counter using Spirit.Lex</a></dt>
+<dt><a href="lex/tutorials/lexer_quickstart3.html"> Quickstart
+ 3 - Counting Words Using a Parser</a></dt>
+</dl></dd>
+<dt> Abstracts</dt>
+<dd><dl>
+<dt>Lexer Primitives</dt>
+<dt><a href="lex/abstracts/lexer_tokenizing.html"> Tokenizing
+ Input Data</a></dt>
+<dt><a href="lex/abstracts/lexer_semantic_actions.html"> Lexer
+ Semantic Actions</a></dt>
+<dt><a href="lex/abstracts/lexer_static_model.html"> The Static
+ Lexer Model</a></dt>
+<dt><a href="lex/abstracts/lexer_parsing.html"> Parsing using
+ a Lexer</a></dt>
+<dt> Lexer Attributes</dt>
+<dt> Lexer States</dt>
+</dl></dd>
+<dt> Quick Reference</dt>
+<dt> Reference</dt>
+<dd><dl>
+<dt> Concepts</dt>
+<dt>Lexer Class</dt>
+<dt>Token Class</dt>
+<dt>TokenDef Class</dt>
+<dt>TokenSet Class</dt>
+</dl></dd>
+</dl></div>
+<div class="section" lang="en">
+<div class="titlepage"><div><div><h3 class="title">
+<a name="spirit.lex.lexer_introduction"></a> Introduction to Spirit.Lex</h3></div></div></div>
+
+ Lexical scanning is the process of analyzing the stream of input characters
+ and separating it into strings called tokens, separated by whitespace. Most
+ compiler texts start here, and devote several chapters to discussing various
+ ways to build scanners. Spirit.Lex is a library built
+ to take care of the complexities of creating a lexer for your grammar (in
+ this documentation we will use the terms 'lexical analyzer', 'lexer' and
+ 'scanner' interchangably). All it needs to create a lexer is to know the
+ set of patterns describing the different tokens you want to recognize in
+ the input. To make this a bit more formal, here are some definitions:
+ 
+<div class="itemizedlist"><ul type="disc">
+<li>
+ A token is a sequence of consecutive characters having a collective meaning.
+ Tokens may have attributes specific to the token type, carrying additional
+ information about the matched character sequence.
+ </li>
+<li>
+ A pattern is a rule expressed as a regular expression and describing how
+ a particular token can be formed. For example, <tt class="literal">[A-Za-z][A-Za-z_0-9]*</tt>
+ is a pattern for a rule matching C++ identifiers.
+ </li>
+<li>
+ Characters between tokens are called whitespace; these include spaces,
+ tabs, newlines, and formfeeds. Many people also count comments as whitespace,
+ though since some tools such as lint look at comments, this conflation
+ is not perfect.
+ </li>
+</ul></div>
+<a name="spirit.lex.lexer_introduction.why_using_a_separate_lexer"></a><h4>
+<a name="id670210"></a>
+ <a href="lex.html#spirit.lex.lexer_introduction.why_using_a_separate_lexer">Why
+ Using a Separate Lexer</a>
+ </h4>
+
+ Typically, lexical scanning is done in a separate module from the parser,
+ feeding the parser with a stream of input tokens only. Now, theoretically
+ it is not necessary to do this separation. In the end there is only one set
+ of syntactical rules defining the language, so in theory we could write the
+ whole parser in one module. In fact, Spirit.Qi allows
+ to write parsers without using a lexer, parsing the input character stream
+ directly, and for the most part this is the way Spirit
+ has been used since its invention.
+ 
+
+ However, the separation has both practical and theoretical bases and proves
+ to be very useful in practical applications. In 1956, Noam Chomsky defined
+ the "Chomsky Hierarchy" of grammars:
+ 
+<div class="itemizedlist"><ul type="disc">
+<li>
+ Type 0: Unrestricted grammars (e.g., natural languages)
+ </li>
+<li>
+ Type 1: Context-Sensitive grammars
+ </li>
+<li>
+ Type 2: Context-Free grammars
+ </li>
+<li>
+ Type 3: Regular grammars
+ </li>
+</ul></div>
+
+ The complexity of these grammars increases from regular grammars being the
+ simplest to unrestricted grammars being the most complex. Similarly, the
+ complexity of the recognizers for these grammars increases. Although, a few
+ features of some programming languages (such as C++) are Type 1, fortunately
+ for the most part programming languages can be described using only the Types
+ 3 and 2. The neat part about these two types is that they are well known
+ and the ways to parse them are well understood. It has been shown that any
+ regular grammar can be parsed using a state machine (finite automaton). Similarly,
+ context-free grammars can always be parsed using a push-down automaton (essentially
+ a state machine augmented by a stack).
+ 
+
+ In real programming languages and practical grammars the parts that can be
+ handled as regular expressions tend to be the lower-level parts, such as
+ the definition of an identifier or of an integer value:
+ 
+<pre class="programlisting">
+letter := [a-zA-Z]
+digit := [0-9]
+
+identifier := letter [ letter | digit ]*
+integer := digit*
+</pre>
+
+ Higher level parts of practical grammars tend to be more complex and can't
+ be implemented using plain regular expressions anymore. We need to store
+ information on the built-in hardware stack while recursing the grammar hierarchy,
+ and that in fact this is the preferred approach used for top-down parsing.
+ Since it takes a different kind of abstract machine to parse the two types
+ of grammars, it proved to be efficient to separate the lexical scanner into
+ a separate module which is built around the idea of a state machine. The
+ goal here is to use the simplest parsing technique needed for the job.
+ 
+
+ Another, more practical reason for separating the scanner from the parser
+ is the need for backtracking during parsing. The input data is a stream of
+ characters, which is often thought to be processed left to right without
+ any backtracking. Unfortunately, in practice most of the time that isn't
+ possible. Almost every language has certain keywords such as IF, FOR, and
+ WHILE. The decision if a certain character sequence actually comprises a
+ keyword or just an identifier often can be made only after seeing the first
+ delimiter after it. This already is a limited form of
+ backtracking, since we need to store the string long enough to be able to
+ make the decision. The same is true for more coarse grained language features
+ such as nested IF/ELSE statements, where the decision about to which IF belongs
+ the last ELSE statement can be made only after seeing the whole construct.
+ 
+
+ So the structure of a conventional compiler often involves splitting up the
+ functions of the lower-level and higher-level parsing. The lexical scanner
+ deals with things at the character level, collecting characters into strings,
+ converting character sequence into different representations as integers,
+ etc., and passing them along to the parser proper as indivisible tokens.
+ It's also considered normal to let the scanner do additional jobs, such as
+ identifying keywords, storing identifiers in tables, etc.
+ 
+
+ Now, Spirit follows this
+ structure, where Spirit.Lex can be used to implement
+ state machine based recognizers, while Spirit.Qi can
+ be used to build recognizers for context free grammars. Since both modules
+ are seemlessly integrated with each other and with the C++ target language
+ it is even possible to use the provided functionality to build more complex
+ grammar recognizers.
+ 
+<a name="spirit.lex.lexer_introduction.advantages_of_using__emphasis_spirit_lex__emphasis_"></a><h4>
+<a name="id670424"></a>
+ <a href="lex.html#spirit.lex.lexer_introduction.advantages_of_using__emphasis_spirit_lex__emphasis_">Advantages
+ of using Spirit.Lex</a>
+ </h4>
+
+ The advantage of using Spirit.Lex to create the lexical
+ analyzer over using more traditional tools such as Flex
+ is its carefully crafted integration with the Spirit
+ library and the C++ host language. You don't need any external tools to generate
+ the code, your lexer will be perfectly integrated with the rest of your program,
+ making it possible to freely access any context information and data structure.
+ Since the C++ compiler sees all the code it will generate optimal code nomatter
+ what configuration options have been chosen by the user. Spirit.Lex
+ gives you all the features you could get from a similar Flex
+ program without the need to leave C++ as a host language:
+ 
+<div class="itemizedlist"><ul type="disc">
+<li>
+ the definition of tokens is done using regular expressions (patterns)
+ </li>
+<li>
+ the token definitions can refer to special substitution string (pattern
+ macros) simplifying pattern definitions
+ </li>
+<li>
+ the generated lexical scanner may have multiple start states
+ </li>
+<li>
+ it is possible to attach code to any of the token definitions; this code
+ gets executed whenever the corresponding token pattern has been matched
+ </li>
+</ul></div>
+
+ Even if it is possible to use Spirit.Lex to generate
+ C++ code representing the lexical analyzer (we will refer to that as the
+ static model, described in more detail in the section
+ <a href="lex/abstracts/lexer_static_model.html" title=" The Static
+ Lexer Model">The Static
+ Model</a>) - a model very similar to the way Flex
+ operates - we will mainly focus on the opposite, the dynamic
+ model. You can directly integrate the token definitions into your C++ program,
+ building the lexical analyzer dynamicly at runtime. The dynamic model is
+ something not supported by Flex
+ or other lexical scanner generators (such as re2c,
+ Ragel, etc.).
+ But it is very flexible and allows to speed up the development of your application.
+ 
+<a name="spirit.lex.lexer_introduction.the_library_structure_of__emphasis_spirit_lex__emphasis_"></a><h4>
+<a name="id670527"></a>
+ <a href="lex.html#spirit.lex.lexer_introduction.the_library_structure_of__emphasis_spirit_lex__emphasis_">The
+ Library Structure of Spirit.Lex</a>
+ </h4>
+
+ The <a href="lex.html#spirit.lexerflow" title="Figure 3. The Library structure and Common Flow of Information while
+ using Spirit.Lex in an application">figure</a> below shows a high level
+ overview of how the Spirit.Lex library might be used
+ in an application. Spirit.Lex allows to create lexical
+ analyzers based on patterns. These patterns are regular expression based
+ rules used to define the different tokens to be recognized in the character
+ input sequence. The input sequence is expected to be provided to the lexical
+ analyzer as an arbitrary standard forward iterator. The lexical analyzer
+ itself exposes a standard forward iterator as well. The difference here is
+ that the exposed iterator provides access to the token sequence instead of
+ to the character sequence. The tokens in this sequence are constructed on
+ the fly by analyzing the underlying character sequence and matching this
+ to the patterns as defined by the application.
+ 
+
+ 
+<div class="figure">
+<a name="spirit.lexerflow"></a>Figure 3. The Library structure and Common Flow of Information while
+ using Spirit.Lex in an application
+<img src=".././images/lexerflow.png" alt="The
+ Library structure and Common Flow of Information while using Spirit.Lex
+ in an application">
+</div>
+
+ 
+</div>
+</div>
+<table xmlns:rev="http://www.cs.rpi.edu/~gregod/boost/tools/doc/revision" width="100%"><tr>
+<td align="left"></td>
+<td align="right"><div class="copyright-footer">Copyright © 2001-2008 Joel de Guzman,
+ Hartmut Kaiser
+ Distributed under the Boost Software License, Version 1.0. (See accompanying
+ file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
+ 
+</div></td>
+</tr></table>
+<hr>
+<div class="spirit-nav">
+<a accesskey="p" href="qi_and_karma/reference/debug.html"><img src="../../../../../doc/html/images/prev.png" alt="Prev"></a><a accesskey="u" href="../index.html"><img src="../../../../../doc/html/images/up.png" alt="Up"></a><a accesskey="h" href="../index.html"><img src="../../../../../doc/html/images/home.png" alt="Home"></a><a accesskey="n" href="lex/tutorials.html"><img src="../../../../../doc/html/images/next.png" alt="Next"></a>
+</div>
+</body>
+</html>

Added: trunk/libs/spirit/doc/html/spirit/lex/abstracts.html
==============================================================================
--- (empty file)
+++ trunk/libs/spirit/doc/html/spirit/lex/abstracts.html 2008-07-14 21:20:34 EDT (Mon, 14 Jul 2008)
@@ -0,0 +1,412 @@
+<html>
+<head>
+<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
+<title> Abstracts</title>
+<link rel="stylesheet" href="../../../../../../doc/html/boostbook.css" type="text/css">
+<meta name="generator" content="DocBook XSL Stylesheets V1.66.1">
+<link rel="start" href="../../index.html" title="Spirit 2.0">
+<link rel="up" href="../lex.html" title=" Spirit.Lex">
+<link rel="prev" href="tutorials/lexer_quickstart3.html" title=" Quickstart
+ 3 - Counting Words Using a Parser">
+<link rel="next" href="abstracts/lexer_tokenizing.html" title=" Tokenizing
+ Input Data">
+</head>
+<body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF">
+<table cellpadding="2" width="100%"><tr>
+<td valign="top"><img alt="Boost C++ Libraries" width="277" height="86" src="../../../../../../boost.png"></td>
+<td align="center">Home</td>
+<td align="center">Libraries</td>
+<td align="center">People</td>
+<td align="center">FAQ</td>
+<td align="center">More</td>
+</tr></table>
+<hr>
+<div class="spirit-nav">
+<a accesskey="p" href="tutorials/lexer_quickstart3.html"><img src="../../../../../../doc/html/images/prev.png" alt="Prev"></a><a accesskey="u" href="../lex.html"><img src="../../../../../../doc/html/images/up.png" alt="Up"></a><a accesskey="h" href="../../index.html"><img src="../../../../../../doc/html/images/home.png" alt="Home"></a><a accesskey="n" href="abstracts/lexer_tokenizing.html"><img src="../../../../../../doc/html/images/next.png" alt="Next"></a>
+</div>
+<div class="section" lang="en">
+<div class="titlepage"><div><div><h3 class="title">
+<a name="spirit.lex.abstracts"></a> Abstracts</h3></div></div></div>
+<div class="toc"><dl>
+<dt>Lexer Primitives</dt>
+<dt><a href="abstracts/lexer_tokenizing.html"> Tokenizing
+ Input Data</a></dt>
+<dt><a href="abstracts/lexer_semantic_actions.html"> Lexer
+ Semantic Actions</a></dt>
+<dt><a href="abstracts/lexer_static_model.html"> The Static
+ Lexer Model</a></dt>
+<dt><a href="abstracts/lexer_parsing.html"> Parsing using
+ a Lexer</a></dt>
+<dt> Lexer Attributes</dt>
+<dt> Lexer States</dt>
+</dl></div>
+<div class="section" lang="en">
+<div class="titlepage"><div><div><h4 class="title">
+<a name="spirit.lex.abstracts.lexer_primitives"></a>Lexer Primitives</h4></div></div></div>
+<div class="toc"><dl>
+<dt><a href="abstracts.html#spirit.lex.abstracts.lexer_primitives.lexer_primitives">
+ Lexer Primitives</a></dt>
+<dt><a href="abstracts.html#spirit.lex.abstracts.lexer_primitives.lexer_token_values">
+ About Tokens and Token Values</a></dt>
+<dt><a href="abstracts.html#spirit.lex.abstracts.lexer_primitives.lexer_token_definition">
+ Ways to define Tokens</a></dt>
+</dl></div>
+<div class="section" lang="en"><div class="titlepage"><div><div><h5 class="title">
+<a name="spirit.lex.abstracts.lexer_primitives.lexer_primitives"></a><a href="abstracts.html#spirit.lex.abstracts.lexer_primitives.lexer_primitives" title="
+ Lexer Primitives">
+ Lexer Primitives</a></h5></div></div></div></div>
+<div class="section" lang="en">
+<div class="titlepage"><div><div><h5 class="title">
+<a name="spirit.lex.abstracts.lexer_primitives.lexer_token_values"></a><a href="abstracts.html#spirit.lex.abstracts.lexer_primitives.lexer_token_values" title="
+ About Tokens and Token Values">
+ About Tokens and Token Values</a></h5></div></div></div>
+
+ As already discussed, lexical scanning is the process of analyzing the
+ stream of input characters and separating it into strings called tokens,
+ most of the time separated by whitespace. The different token types recognized
+ by a lexical analyzer often get assigned unique integer token identifiers
+ (token ids). These token ids arenormally used by the parser to identifiy
+ the current token without having to look at the matched string again.
+ The Spirit.Lex library is not different with respect
+ to this, as it uses the token ids as the main means of identification
+ of the different token types defined for a particular lexical analyzer.
+ However, it is different from commonly used lexical analyzers in the
+ sense that it returns (references to) instances of a (user defined) token
+ class to the user. The only real limitation posed on this token class
+ is consequently, that it has to carry at least the token id of the token
+ it represents. For more information about the interface a user defined
+ token type has to expose please look at the <a href="reference.html#spirit.lex.reference.concepts.token" title="Token">Token
+ Reference</a> reference. The library provides a default token type
+ based on the Lexertl
+ library which should be sufficient in most use cases: the lexertl_token<>
+ type. This section focusses on the description of general features a
+ token class may implement and how this integrates with the other parts
+ of the Spirit.Lex library.
+ 
+<a name="spirit.lex.abstracts.lexer_primitives.lexer_token_values.the_anatomy_of_a_token"></a><h6>
+<a name="id680780"></a>
+ <a href="abstracts.html#spirit.lex.abstracts.lexer_primitives.lexer_token_values.the_anatomy_of_a_token">The
+ Anatomy of a Token</a>
+ </h6>
+
+ It is very important to understand the difference between a token definition
+ (represented by the token_def<>
+ template) and a token itself (for instance represented by the lexertl_token<>
+ template).
+ 
+
+ The token definition is used to describe the main features of a particular
+ token type, especially:
+ 
+<div class="itemizedlist"><ul type="disc">
+<li>
+ to simplify the definition of a token type using a regular expression
+ pattern applied while matching this token type,
+ </li>
+<li>
+ to associate a token type with a particular lexer state,
+ </li>
+<li>
+ to optionally assign a token id to a token type,
+ </li>
+<li>
+ to optionally associate some code to execute whenever an instance of
+ this token type has been matched,
+ </li>
+<li>
+ and to optionally specify the attribute type of the token value.
+ </li>
+</ul></div>
+
+ The token itself is a data structure returned by the lexer iterators.
+ Dereferencing a lexer iterator returns a reference to the last matched
+ token instance. It encapsulates the part of the underlying input sequence
+ matched by the regular expression used during the definiton of this token
+ type. Incrementing the lexer iterator invokes the lexical analyzer to
+ match the next token by advancing the underlying input stream. The token
+ data structure contains at least the token id of the matched token type,
+ allowing to identify the matched character sequence. Optionally, the
+ token instance may contain a token value and/or the lexer state this
+ token instance was matched in. The following figure
+ shows the schematic structure of a token.
+ 
+
+ 
+<div class="figure">
+<a name="spirit.lex.tokenstructure"></a>Figure 5. The structure of a token
+<img src="../.././images/TokenStructure.png" alt="The
+ structure of a token">
+</div>
+
+ 
+
+ The token value and the token state may be omitted for optimization reasons,
+ avoiding the token to carry more data than actually required. This configuration
+ can be achieved by supplying appropriate template parameters for the
+ lexertl_token<> template while defining
+ the token type.
+ 
+
+ The lexer iterator returns the same token type for each of the different
+ matched token definitions. To accomodate for the possibly different token
+ value types exposed by the various token types (token
+ definitions), the general type of the token value is a boost.variant<>.
+ As a minimum (for the default configuration) this token value variant
+ will be configured to always hold a boost::iterator_range<>
+ containing the pair of iterators pointing to the matched input sequence
+ for this token instance.
+ 
+<div class="note"><table border="0" summary="Note">
+<tr>
+<td rowspan="2" align="center" valign="top" width="25"><img alt="[Note]" src="../../images/note.png"></td>
+<th align="left">Note</th>
+</tr>
+<tr><td colspan="2" align="left" valign="top">
+ If the lexical analyzer is used in conjunction with a Spirit.Qi
+ parser, the stored boost::iterator_range<>
+ token value will be converted to the requested token type (parser attribute)
+ exactly once. This happens at the time of the first access to the token
+ value requiring the corresponding type conversion. The converted token
+ value will be stored in the boost.variant<>
+ replacing the initially stored iterator range. This avoids to convert
+ the input sequence to the token value more than once, thus optimizing
+ the integration of the lexer with Spirit.Qi, even
+ during parser backtracking.
+ </td></tr>
+</table></div>
+
+ Here is the template prototype of the lexertl_token<>
+ template:
+ 
+<pre class="programlisting">
+template <
+ typename Iterator = char const*,
+ typename AttributeTypes = mpl::vector0<>,
+ typename HasState = mpl::true_
+>
+struct lexertl_token;
+</pre>
+<div class="variablelist">
+where:
+<dl>
+<dt>Iterator</dt>
+<dd>
+ This is the type of the iterator used to access the underlying input
+ stream. It defaults to a plain <tt class="computeroutput">char
+ const*</tt>.
+ </dd>
+<dt>AttributeTypes</dt>
+<dd>
+ This is either a mpl sequence containing all attribute types used
+ for the token definitions or the type <tt class="computeroutput">omitted</tt>.
+ If the mpl sequence is empty (which is the default), all token instances
+ will store a <tt class="computeroutput">boost::iterator_range<Iterator></tt> pointing to the start and the
+ end of the matched section in the input stream. If the type is <tt class="computeroutput">omitted</tt>, the generated tokens will
+ contain no token value (attribute) at all.
+ </dd>
+<dt>HasState</dt>
+<dd>
+ This is either <tt class="computeroutput">mpl::true_</tt>
+ or <tt class="computeroutput">mpl::false_</tt>, allowing to control whether
+ the generated token instances will contain the lexer state they were
+ generated in. The default is mpl::true_, so all token instances will
+ contain the lexer state.
+ </dd>
+</dl>
+</div>
+
+ Normally, during its construction, a token instance always holds the
+ boost::iterator_range<>
+ as its token value (except, if it has been defined using the <tt class="computeroutput">omitted</tt> token value type). This iterator
+ range then is converted in place to the requested token value type (attribute)
+ when it is requested for the first time.
+ 
+<a name="spirit.lex.abstracts.lexer_primitives.lexer_token_values.the_physiognomy_of_a_token_definition"></a><h6>
+<a name="id681307"></a>
+ <a href="abstracts.html#spirit.lex.abstracts.lexer_primitives.lexer_token_values.the_physiognomy_of_a_token_definition">The
+ Physiognomy of a Token Definition</a>
+ </h6>
+
+ The token definitions (represented by the token_def<>
+ template) are normally used as part of the definition of the lexical
+ analyzer. At the same time a token definition instance may be used as
+ a parser component in Spirit.Qi.
+ 
+
+ The template prototype of this class is shown here:
+ 
+<pre class="programlisting">
+template<
+ typename Attribute = unused_type,
+ typename Char = char
+>
+class token_def;
+</pre>
+<div class="variablelist">
+where:
+<dl>
+<dt>Attribute</dt>
+<dd>
+ This is the type of the token value (attribute) supported by token
+ instances representing this token type. This attribute type is exposed
+ to the Spirit.Qi library, whenever this token
+ definition is used as a parser component. The default attribute type
+ is <tt class="computeroutput">unused_type</tt>, which
+ means the token instance holds a boost::iterator_range<>
+ pointing to the start and the end of the matched section in the input
+ stream. If the attribute is <tt class="computeroutput">omitted</tt>
+ the token instance will expose no token type at all. Any other type
+ will be used directly as the token value type.
+ </dd>
+<dt>Char</dt>
+<dd>
+ This is the value type of the iterator for the underlying input sequence.
+ It defaults to <tt class="computeroutput">char</tt>.
+ </dd>
+</dl>
+</div>
+
+ The semantics of the template parameters for the token type and the token
+ definition type are very similar and interdependent. As a rule of thumb
+ you can think of the token definition type as the means of specifying
+ everything related to a single specific token type (such as <tt class="computeroutput">identifier</tt> or <tt class="computeroutput">integer</tt>).
+ On the other hand the token type is used to define the general proerties
+ of all token instances generated by the Spirit.Lex
+ library.
+ 
+<div class="important"><table border="0" summary="Important">
+<tr>
+<td rowspan="2" align="center" valign="top" width="25"><img alt="[Important]" src="../../images/important.png"></td>
+<th align="left">Important</th>
+</tr>
+<tr><td colspan="2" align="left" valign="top">
+
+ If you don't list any token value types in the token type definition
+ declaration (resulting in the usage of the default boost::iterator_range<>
+ token type) everything will compile and work just fine, just a bit
+ less efficient. This is because the token value will be converted from
+ the matched input sequence every time it is requested.
+ 
+
+ But as soon as you specify at least one token value type while defining
+ the token type you'll have to list all value types used for token_def<>
+ declarations in the token definition class, otherwise compilation errors
+ will occur.
+ 
+</td></tr>
+</table></div>
+<a name="spirit.lex.abstracts.lexer_primitives.lexer_token_values.examples_of_using__link_linkend__spirit_lex_reference_token_class___code__phrase_role__identifier__lexertl_token__phrase__phrase_role__special___lt__gt___phrase___code___link_"></a><h6>
+<a name="id681564"></a>
+ <a href="abstracts.html#spirit.lex.abstracts.lexer_primitives.lexer_token_values.examples_of_using__link_linkend__spirit_lex_reference_token_class___code__phrase_role__identifier__lexertl_token__phrase__phrase_role__special___lt__gt___phrase___code___link_">Examples
+ of using lexertl_token<></a>
+ </h6>
+
+ Let's start with some examples. We refer to one of the Spirit.Lex
+ examples (for the full source code of this example please see example4.cpp).
+ 
+
+ The first code snippet shows an excerpt of the token definition class,
+ the definition of a couple of token types. Some of the token types do
+ not expose a special token value (<tt class="computeroutput">if_</tt>,
+ <tt class="computeroutput">else_</tt>, and <tt class="computeroutput">while_</tt>). Their token value will always
+ hold the iterator range of the matched input sequence only. The token
+ definitions for the <tt class="computeroutput">identifier</tt>
+ and the integer <tt class="computeroutput">constant</tt>
+ are specialized to expose an explicit token type each: <tt class="computeroutput">std::string</tt> and <tt class="computeroutput">unsigned
+ int</tt>.
+ 
+
+ 
+
+
+
+<pre class="programlisting">
+// these tokens expose the iterator_range of the matched input sequence
+token_def<> if_, else_, while_;
+
+// The following two tokens have an associated value type, 'identifier'
+// carries a string (the identifier name) and 'constant' carries the
+// matched integer value.
+//
+// Note: any token value type specified explicitly during a token_def<>
+// declaration needs to be listed during token type definition as
+// well (see the typedef for the token_type below).
+//
+// The conversion of the matched input to an instance of this type occurs
+// once (on first access), which makes token values as efficient as
+// possible. Moreover, token instances are constructed once by the lexer
+// library. From this point on tokens are passed by reference only,
+// avoiding tokens being copied around.
+token_def<std::string> identifier;
+token_def<unsigned int> constant;
+</pre>
+
+ 
+
+ 
+
+ As the parsers generated by Spirit.Qi are fully
+ attributed, any Spirit.Qi parser component needs
+ to expose a certain type as its parser attribute. Naturally, the token_def<> exposes the token value
+ type as its parser attribute, enabling a smooth integration with Spirit.Qi.
+ 
+
+ The next code snippet demonstrates how the required token value types
+ are specified while defining the token type to use. All of the token
+ value types used for at least one of the token definitions have to be
+ re-iterated for the token definition as well.
+ 
+
+ 
+
+
+
+<pre class="programlisting">
+// This is the lexer token type to use. The second template parameter lists
+// all attribute types used for token_def's during token definition (see
+// calculator_tokens<> above). Here we use the predefined lexertl token
+// type, but any compatible token type may be used instead.
+//
+// If you don't list any token value types in the following declaration
+// (or just use the default token type: lexertl_token<base_iterator_type>)
+// it will compile and work just fine, just a bit less efficient. This is
+// because the token value will be generated from the matched input
+// sequence every time it is requested. But as soon as you specify at
+// least one token value type you'll have to list all value types used
+// for token_def<> declarations in the token definition class above,
+// otherwise compilation errors will occur.
+typedef lexertl_token<
+ base_iterator_type, boost::mpl::vector<unsigned int, std::string>
+> token_type;
+</pre>
+
+ 
+
+ 
+
+ To avoid the token to have a token value at all, the special tag <tt class="computeroutput">omitted</tt> can be used: <tt class="computeroutput">token_def<omitted></tt>
+ and <tt class="computeroutput">lexertl_token<base_iterator_type,
+ omitted></tt>.
+ 
+</div>
+<div class="section" lang="en"><div class="titlepage"><div><div><h5 class="title">
+<a name="spirit.lex.abstracts.lexer_primitives.lexer_token_definition"></a><a href="abstracts.html#spirit.lex.abstracts.lexer_primitives.lexer_token_definition" title="
+ Ways to define Tokens">
+ Ways to define Tokens</a></h5></div></div></div></div>
+</div>
+</div>
+<table xmlns:rev="http://www.cs.rpi.edu/~gregod/boost/tools/doc/revision" width="100%"><tr>
+<td align="left"></td>
+<td align="right"><div class="copyright-footer">Copyright © 2001-2008 Joel de Guzman,
+ Hartmut Kaiser
+ Distributed under the Boost Software License, Version 1.0. (See accompanying
+ file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
+ 
+</div></td>
+</tr></table>
+<hr>
+<div class="spirit-nav">
+<a accesskey="p" href="tutorials/lexer_quickstart3.html"><img src="../../../../../../doc/html/images/prev.png" alt="Prev"></a><a accesskey="u" href="../lex.html"><img src="../../../../../../doc/html/images/up.png" alt="Up"></a><a accesskey="h" href="../../index.html"><img src="../../../../../../doc/html/images/home.png" alt="Home"></a><a accesskey="n" href="abstracts/lexer_tokenizing.html"><img src="../../../../../../doc/html/images/next.png" alt="Next"></a>
+</div>
+</body>
+</html>

Added: trunk/libs/spirit/doc/html/spirit/lex/abstracts/lexer_attributes.html
==============================================================================
--- (empty file)
+++ trunk/libs/spirit/doc/html/spirit/lex/abstracts/lexer_attributes.html 2008-07-14 21:20:34 EDT (Mon, 14 Jul 2008)
@@ -0,0 +1,42 @@
+<html>
+<head>
+<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
+<title> Lexer Attributes</title>
+<link rel="stylesheet" href="../../../../../../../doc/html/boostbook.css" type="text/css">
+<meta name="generator" content="DocBook XSL Stylesheets V1.66.1">
+<link rel="start" href="../../../index.html" title="Spirit 2.0">
+<link rel="up" href="../abstracts.html" title=" Abstracts">
+<link rel="prev" href="lexer_parsing.html" title=" Parsing using
+ a Lexer">
+<link rel="next" href="lexer_states.html" title=" Lexer States">
+</head>
+<body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF">
+<table cellpadding="2" width="100%"><tr>
+<td valign="top"><img alt="Boost C++ Libraries" width="277" height="86" src="../../../../../../../boost.png"></td>
+<td align="center">Home</td>
+<td align="center">Libraries</td>
+<td align="center">People</td>
+<td align="center">FAQ</td>
+<td align="center">More</td>
+</tr></table>
+<hr>
+<div class="spirit-nav">
+<a accesskey="p" href="lexer_parsing.html"><img src="../../../../../../../doc/html/images/prev.png" alt="Prev"></a><a accesskey="u" href="../abstracts.html"><img src="../../../../../../../doc/html/images/up.png" alt="Up"></a><a accesskey="h" href="../../../index.html"><img src="../../../../../../../doc/html/images/home.png" alt="Home"></a><a accesskey="n" href="lexer_states.html"><img src="../../../../../../../doc/html/images/next.png" alt="Next"></a>
+</div>
+<div class="section" lang="en"><div class="titlepage"><div><div><h4 class="title">
+<a name="spirit.lex.abstracts.lexer_attributes"></a> Lexer Attributes</h4></div></div></div></div>
+<table xmlns:rev="http://www.cs.rpi.edu/~gregod/boost/tools/doc/revision" width="100%"><tr>
+<td align="left"></td>
+<td align="right"><div class="copyright-footer">Copyright © 2001-2008 Joel de Guzman,
+ Hartmut Kaiser
+ Distributed under the Boost Software License, Version 1.0. (See accompanying
+ file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
+ 
+</div></td>
+</tr></table>
+<hr>
+<div class="spirit-nav">
+<a accesskey="p" href="lexer_parsing.html"><img src="../../../../../../../doc/html/images/prev.png" alt="Prev"></a><a accesskey="u" href="../abstracts.html"><img src="../../../../../../../doc/html/images/up.png" alt="Up"></a><a accesskey="h" href="../../../index.html"><img src="../../../../../../../doc/html/images/home.png" alt="Home"></a><a accesskey="n" href="lexer_states.html"><img src="../../../../../../../doc/html/images/next.png" alt="Next"></a>
+</div>
+</body>
+</html>

Added: trunk/libs/spirit/doc/html/spirit/lex/abstracts/lexer_parsing.html
==============================================================================
--- (empty file)
+++ trunk/libs/spirit/doc/html/spirit/lex/abstracts/lexer_parsing.html 2008-07-14 21:20:34 EDT (Mon, 14 Jul 2008)
@@ -0,0 +1,45 @@
+<html>
+<head>
+<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
+<title> Parsing using
+ a Lexer</title>
+<link rel="stylesheet" href="../../../../../../../doc/html/boostbook.css" type="text/css">
+<meta name="generator" content="DocBook XSL Stylesheets V1.66.1">
+<link rel="start" href="../../../index.html" title="Spirit 2.0">
+<link rel="up" href="../abstracts.html" title=" Abstracts">
+<link rel="prev" href="lexer_static_model.html" title=" The Static
+ Lexer Model">
+<link rel="next" href="lexer_attributes.html" title=" Lexer Attributes">
+</head>
+<body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF">
+<table cellpadding="2" width="100%"><tr>
+<td valign="top"><img alt="Boost C++ Libraries" width="277" height="86" src="../../../../../../../boost.png"></td>
+<td align="center">Home</td>
+<td align="center">Libraries</td>
+<td align="center">People</td>
+<td align="center">FAQ</td>
+<td align="center">More</td>
+</tr></table>
+<hr>
+<div class="spirit-nav">
+<a accesskey="p" href="lexer_static_model.html"><img src="../../../../../../../doc/html/images/prev.png" alt="Prev"></a><a accesskey="u" href="../abstracts.html"><img src="../../../../../../../doc/html/images/up.png" alt="Up"></a><a accesskey="h" href="../../../index.html"><img src="../../../../../../../doc/html/images/home.png" alt="Home"></a><a accesskey="n" href="lexer_attributes.html"><img src="../../../../../../../doc/html/images/next.png" alt="Next"></a>
+</div>
+<div class="section" lang="en"><div class="titlepage"><div><div><h4 class="title">
+<a name="spirit.lex.abstracts.lexer_parsing"></a><a href="lexer_parsing.html" title=" Parsing using
+ a Lexer"> Parsing using
+ a Lexer</a></h4></div></div></div></div>
+<table xmlns:rev="http://www.cs.rpi.edu/~gregod/boost/tools/doc/revision" width="100%"><tr>
+<td align="left"></td>
+<td align="right"><div class="copyright-footer">Copyright © 2001-2008 Joel de Guzman,
+ Hartmut Kaiser
+ Distributed under the Boost Software License, Version 1.0. (See accompanying
+ file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
+ 
+</div></td>
+</tr></table>
+<hr>
+<div class="spirit-nav">
+<a accesskey="p" href="lexer_static_model.html"><img src="../../../../../../../doc/html/images/prev.png" alt="Prev"></a><a accesskey="u" href="../abstracts.html"><img src="../../../../../../../doc/html/images/up.png" alt="Up"></a><a accesskey="h" href="../../../index.html"><img src="../../../../../../../doc/html/images/home.png" alt="Home"></a><a accesskey="n" href="lexer_attributes.html"><img src="../../../../../../../doc/html/images/next.png" alt="Next"></a>
+</div>
+</body>
+</html>

Added: trunk/libs/spirit/doc/html/spirit/lex/abstracts/lexer_semantic_actions.html
==============================================================================
--- (empty file)
+++ trunk/libs/spirit/doc/html/spirit/lex/abstracts/lexer_semantic_actions.html 2008-07-14 21:20:34 EDT (Mon, 14 Jul 2008)
@@ -0,0 +1,46 @@
+<html>
+<head>
+<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
+<title> Lexer
+ Semantic Actions</title>
+<link rel="stylesheet" href="../../../../../../../doc/html/boostbook.css" type="text/css">
+<meta name="generator" content="DocBook XSL Stylesheets V1.66.1">
+<link rel="start" href="../../../index.html" title="Spirit 2.0">
+<link rel="up" href="../abstracts.html" title=" Abstracts">
+<link rel="prev" href="lexer_tokenizing.html" title=" Tokenizing
+ Input Data">
+<link rel="next" href="lexer_static_model.html" title=" The Static
+ Lexer Model">
+</head>
+<body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF">
+<table cellpadding="2" width="100%"><tr>
+<td valign="top"><img alt="Boost C++ Libraries" width="277" height="86" src="../../../../../../../boost.png"></td>
+<td align="center">Home</td>
+<td align="center">Libraries</td>
+<td align="center">People</td>
+<td align="center">FAQ</td>
+<td align="center">More</td>
+</tr></table>
+<hr>
+<div class="spirit-nav">
+<a accesskey="p" href="lexer_tokenizing.html"><img src="../../../../../../../doc/html/images/prev.png" alt="Prev"></a><a accesskey="u" href="../abstracts.html"><img src="../../../../../../../doc/html/images/up.png" alt="Up"></a><a accesskey="h" href="../../../index.html"><img src="../../../../../../../doc/html/images/home.png" alt="Home"></a><a accesskey="n" href="lexer_static_model.html"><img src="../../../../../../../doc/html/images/next.png" alt="Next"></a>
+</div>
+<div class="section" lang="en"><div class="titlepage"><div><div><h4 class="title">
+<a name="spirit.lex.abstracts.lexer_semantic_actions"></a><a href="lexer_semantic_actions.html" title=" Lexer
+ Semantic Actions"> Lexer
+ Semantic Actions</a></h4></div></div></div></div>
+<table xmlns:rev="http://www.cs.rpi.edu/~gregod/boost/tools/doc/revision" width="100%"><tr>
+<td align="left"></td>
+<td align="right"><div class="copyright-footer">Copyright © 2001-2008 Joel de Guzman,
+ Hartmut Kaiser
+ Distributed under the Boost Software License, Version 1.0. (See accompanying
+ file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
+ 
+</div></td>
+</tr></table>
+<hr>
+<div class="spirit-nav">
+<a accesskey="p" href="lexer_tokenizing.html"><img src="../../../../../../../doc/html/images/prev.png" alt="Prev"></a><a accesskey="u" href="../abstracts.html"><img src="../../../../../../../doc/html/images/up.png" alt="Up"></a><a accesskey="h" href="../../../index.html"><img src="../../../../../../../doc/html/images/home.png" alt="Home"></a><a accesskey="n" href="lexer_static_model.html"><img src="../../../../../../../doc/html/images/next.png" alt="Next"></a>
+</div>
+</body>
+</html>

Added: trunk/libs/spirit/doc/html/spirit/lex/abstracts/lexer_states.html
==============================================================================
--- (empty file)
+++ trunk/libs/spirit/doc/html/spirit/lex/abstracts/lexer_states.html 2008-07-14 21:20:34 EDT (Mon, 14 Jul 2008)
@@ -0,0 +1,58 @@
+<html>
+<head>
+<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
+<title> Lexer States</title>
+<link rel="stylesheet" href="../../../../../../../doc/html/boostbook.css" type="text/css">
+<meta name="generator" content="DocBook XSL Stylesheets V1.66.1">
+<link rel="start" href="../../../index.html" title="Spirit 2.0">
+<link rel="up" href="../abstracts.html" title=" Abstracts">
+<link rel="prev" href="lexer_attributes.html" title=" Lexer Attributes">
+<link rel="next" href="../quick_reference.html" title=" Quick Reference">
+</head>
+<body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF">
+<table cellpadding="2" width="100%"><tr>
+<td valign="top"><img alt="Boost C++ Libraries" width="277" height="86" src="../../../../../../../boost.png"></td>
+<td align="center">Home</td>
+<td align="center">Libraries</td>
+<td align="center">People</td>
+<td align="center">FAQ</td>
+<td align="center">More</td>
+</tr></table>
+<hr>
+<div class="spirit-nav">
+<a accesskey="p" href="lexer_attributes.html"><img src="../../../../../../../doc/html/images/prev.png" alt="Prev"></a><a accesskey="u" href="../abstracts.html"><img src="../../../../../../../doc/html/images/up.png" alt="Up"></a><a accesskey="h" href="../../../index.html"><img src="../../../../../../../doc/html/images/home.png" alt="Home"></a><a accesskey="n" href="../quick_reference.html"><img src="../../../../../../../doc/html/images/next.png" alt="Next"></a>
+</div>
+<div class="section" lang="en">
+<div class="titlepage"><div><div><h4 class="title">
+<a name="spirit.lex.abstracts.lexer_states"></a> Lexer States</h4></div></div></div>
+<a name="spirit.lex.abstracts.lexer_states.controlling_the_lexer_state_from_lexer_semantic_actions"></a><h5>
+<a name="id686439"></a>
+ <a href="lexer_states.html#spirit.lex.abstracts.lexer_states.controlling_the_lexer_state_from_lexer_semantic_actions">Controlling
+ the Lexer State from Lexer Semantic Actions</a>
+ </h5>
+<a name="spirit.lex.abstracts.lexer_states.controlling_the_lexer_state_from_parser_semantic_actions"></a><h5>
+<a name="id686452"></a>
+ <a href="lexer_states.html#spirit.lex.abstracts.lexer_states.controlling_the_lexer_state_from_parser_semantic_actions">Controlling
+ the Lexer State from Parser Semantic Actions</a>
+ </h5>
+<a name="spirit.lex.abstracts.lexer_states.using_a_lexer_state_for_the_skip_parser"></a><h5>
+<a name="id686465"></a>
+ <a href="lexer_states.html#spirit.lex.abstracts.lexer_states.using_a_lexer_state_for_the_skip_parser">Using
+ a Lexer State for the Skip Parser</a>
+ </h5>
+</div>
+<table xmlns:rev="http://www.cs.rpi.edu/~gregod/boost/tools/doc/revision" width="100%"><tr>
+<td align="left"></td>
+<td align="right"><div class="copyright-footer">Copyright © 2001-2008 Joel de Guzman,
+ Hartmut Kaiser
+ Distributed under the Boost Software License, Version 1.0. (See accompanying
+ file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
+ 
+</div></td>
+</tr></table>
+<hr>
+<div class="spirit-nav">
+<a accesskey="p" href="lexer_attributes.html"><img src="../../../../../../../doc/html/images/prev.png" alt="Prev"></a><a accesskey="u" href="../abstracts.html"><img src="../../../../../../../doc/html/images/up.png" alt="Up"></a><a accesskey="h" href="../../../index.html"><img src="../../../../../../../doc/html/images/home.png" alt="Home"></a><a accesskey="n" href="../quick_reference.html"><img src="../../../../../../../doc/html/images/next.png" alt="Next"></a>
+</div>
+</body>
+</html>

Added: trunk/libs/spirit/doc/html/spirit/lex/abstracts/lexer_static_model.html
==============================================================================
--- (empty file)
+++ trunk/libs/spirit/doc/html/spirit/lex/abstracts/lexer_static_model.html 2008-07-14 21:20:34 EDT (Mon, 14 Jul 2008)
@@ -0,0 +1,337 @@
+<html>
+<head>
+<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
+<title> The Static
+ Lexer Model</title>
+<link rel="stylesheet" href="../../../../../../../doc/html/boostbook.css" type="text/css">
+<meta name="generator" content="DocBook XSL Stylesheets V1.66.1">
+<link rel="start" href="../../../index.html" title="Spirit 2.0">
+<link rel="up" href="../abstracts.html" title=" Abstracts">
+<link rel="prev" href="lexer_semantic_actions.html" title=" Lexer
+ Semantic Actions">
+<link rel="next" href="lexer_parsing.html" title=" Parsing using
+ a Lexer">
+</head>
+<body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF">
+<table cellpadding="2" width="100%"><tr>
+<td valign="top"><img alt="Boost C++ Libraries" width="277" height="86" src="../../../../../../../boost.png"></td>
+<td align="center">Home</td>
+<td align="center">Libraries</td>
+<td align="center">People</td>
+<td align="center">FAQ</td>
+<td align="center">More</td>
+</tr></table>
+<hr>
+<div class="spirit-nav">
+<a accesskey="p" href="lexer_semantic_actions.html"><img src="../../../../../../../doc/html/images/prev.png" alt="Prev"></a><a accesskey="u" href="../abstracts.html"><img src="../../../../../../../doc/html/images/up.png" alt="Up"></a><a accesskey="h" href="../../../index.html"><img src="../../../../../../../doc/html/images/home.png" alt="Home"></a><a accesskey="n" href="lexer_parsing.html"><img src="../../../../../../../doc/html/images/next.png" alt="Next"></a>
+</div>
+<div class="section" lang="en">
+<div class="titlepage"><div><div><h4 class="title">
+<a name="spirit.lex.abstracts.lexer_static_model"></a><a href="lexer_static_model.html" title=" The Static
+ Lexer Model"> The Static
+ Lexer Model</a></h4></div></div></div>
+
+ The documentation of Spirit.Lex so far mostly was
+ about describing the features of the dynamic model,
+ where the tables needed for lexical analysis are generated from the regular
+ expressions at runtime. The big advantage of the dynamic model is its flexibility,
+ and its integration with the Spirit
+ library and the C++ host language. Its big disadvantage is the need to
+ spend additional runtime to generate the tables, which especially might
+ be a limitation for larger lexical analyers. The static
+ model strives to build upon the smooth integration with Spirit
+ and C++, and reuses large parts of the Spirit.Lex
+ library as described so far, while overcoming the additional runtime requirements
+ by using pre-generated tables and tokenizer routines. To make the code
+ generation as simple as possible, it is possible reuse the token definition
+ types developed using the dynamic model without any
+ changes. As will be shown in this section, building a code generator based
+ on an existing token definition type is a matter of writing 3 lines of
+ code.
+ 
+
+ Assuming you already built a dynamic lexer for your problem, there are
+ two more steps needed to create a static lexical analyzer using Spirit.Lex:
+ 
+<div class="orderedlist"><ol type="1">
+<li>
+ generating the C++ code for the static analyzer (including the tokenization
+ function and corresponding tables), and
+ </li>
+<li>
+ modifying the dynamic lexical anlyzer to use the generated code.
+ </li>
+</ol></div>
+
+ Both steps are described in more detail in the two sections below (for
+ the full source code used in this example see the code here: <a href="../../../../../example/lex/static_lexer/word_count_tokens.hpp" target="_top">the
+ common token definition</a>, <a href="../../../../../example/lex/static_lexer/word_count_generate.cpp" target="_top">the
+ code generator</a>, <a href="../../../../../example/lex/static_lexer/word_count_static.hpp" target="_top">the
+ generated code</a>, and <a href="../../../../../example/lex/static_lexer/word_count_static.cpp" target="_top">the
+ static lexical analyzer</a>).
+ 
+
+ But first we provide the code snippets needed to understand the further
+ descriptions. Both, the definition of the used token identifier and the
+ of the token definition class in this example are put into a separate header
+ file to make these available to the code generator and the static lexical
+ analyzer.
+ 
+
+ 
+
+
+
+<pre class="programlisting">
+enum tokenids
+{
+ IDANY = boost::spirit::lex::min_token_id + 1,
+};
+</pre>
+
+ 
+
+ 
+
+ The important point here is, that the token definition class is not different
+ from a similar class to be used for a dynamic lexical analyzer. The library
+ has been designed in a way, that all components (dynamic lexical analyzer,
+ code generator, and static lexical analyzer) can reuse the very same token
+ definition syntax.
+ 
+
+ 
+
+
+
+<pre class="programlisting">
+// This token definition class can be used without any change for all three
+// possible use cases: a dynamic lexical analyzer, a code generator, and a
+// static lexical analyzer.
+template <typename BaseLexer>
+struct word_count_tokens : boost::spirit::lex::lexer_def<BaseLexer>
+{
+ template <typename Self>
+ void def (Self& self)
+ {
+ // define tokens and associate them with the lexer
+ word = "[^ \t\n]+";
+ self = word | '\n' | token_def<>(".", IDANY);
+ }
+
+ boost::spirit::lex::token_def<std::string> word;
+};
+</pre>
+
+ 
+
+ 
+
+ The only thing changing between the three different use cases is the template
+ parameter used to instantiate a concrete token definition. Fot the dynamic
+ model and the code generator you probably will use the lexertl_lexer<>
+ template, where for the static model you will use the lexertl_static_lexer<>
+ type as the template parameter.
+ 
+
+ This example not only shows how to build a static lexer, but it additionally
+ demonstrates, how such a lexer can be used for parsing in conjunction with
+ a Spirit.Qi grammar. For completeness we provide the
+ simple grammar used in this example. As you can see, this grammar does
+ not have any dependencies on the static lexical analyzer, and for this
+ reason it is not different from a grammar used either without a lexer or
+ using a dynamic lexical analyzer as described before.
+ 
+
+ 
+
+
+
+<pre class="programlisting">
+// This is an ordinary grammar definition following the rules defined by
+// Spirit.Qi. There is nothing specific about it, except it gets the token
+// definition class instance passed to the constructor to allow accessing the
+// embedded token_def<> instances.
+template <typename Iterator>
+struct word_count_grammar : grammar<Iterator>
+{
+ template <typename TokenDef>
+ word_count_grammar(TokenDef const& tok)
+ : grammar<Iterator>(start), c(0), w(0), l(0)
+ {
+ using boost::spirit::arg_names::_1;
+ using boost::phoenix::ref;
+ using boost::phoenix::size;
+
+ // associate the defined tokens with the lexer, at the same time
+ // defining the actions to be executed
+ start = *( tok.word [++ref(w), ref(c) += size(_1)]
+ | char_('\n') [++ref(l), ++ref(c)]
+ | token(IDANY) [++ref(c)]
+ )
+ ;
+ }
+
+ std::size_t c, w, l; // counter for characters, words, and lines
+ rule<Iterator> start;
+};
+</pre>
+
+ 
+
+ 
+<a name="spirit.lex.abstracts.lexer_static_model.generating_the_static_analyzer"></a><h5>
+<a name="id683895"></a>
+ <a href="lexer_static_model.html#spirit.lex.abstracts.lexer_static_model.generating_the_static_analyzer">Generating
+ the Static Analyzer</a>
+ </h5>
+
+ The first additional step to perform in order to create a static lexical
+ analyzer is to create a small standalone program for creating the lexer
+ tables and the corresponding tokenization function. For this purpose the
+ Spirit.Lex library exposes a special API - the function
+ generate_static().
+ It implements the whole code generator, no further code is needed. All
+ what it takes to invoke this function is to supply a token definition instance,
+ an output stream to use to generate the code to, and an optional string
+ to be used as a prefix for the name of the generated function. All in all
+ just a couple lines of code.
+ 
+
+ 
+
+
+
+<pre class="programlisting">
+int main(int argc, char* argv[])
+{
+ // create the lexer object instance needed to invoke the generator
+ word_count_tokens<lexertl_lexer<> > word_count; // the token definition
+
+ // open the output file, where the generated tokenizer function will be
+ // written to
+ std::ofstream out(argc < 2 ? "word_count_static.hpp" : argv[1]);
+
+ // invoke the generator, passing the token definition, the output stream
+ // and the name prefix of the tokenizing function to be generated
+ char const* function_name = (argc < 3 ? "" : argv[2]);
+ return generate_static(make_lexer(word_count), out, function_name) ? 0 : -1;
+}
+</pre>
+
+ 
+
+ 
+
+ The shown code generator will generate output, which should be stored in
+ a file for later inclusion into the static lexical analzyer as shown in
+ the next topic (the full generated code can be viewed here).
+ 
+<a name="spirit.lex.abstracts.lexer_static_model.modifying_the_dynamic_analyzer"></a><h5>
+<a name="id684265"></a>
+ <a href="lexer_static_model.html#spirit.lex.abstracts.lexer_static_model.modifying_the_dynamic_analyzer">Modifying
+ the Dynamic Analyzer</a>
+ </h5>
+
+ The second required step to convert an existing dynamic lexer into a static
+ one is to change your main program at two places. First, you need to change
+ the type of the used lexer (that is the template parameter used while instantiating
+ your token definition class). While in the dynamic model we have been using
+ the lexertl_lexer<>
+ template, we now need to change that to the lexertl_static_lexer<>
+ type. The second change is tightly related to the first one and involves
+ correcting the corresponding <tt class="computeroutput">#include</tt>
+ statement to:
+ 
+
+ 
+
+
+
+<pre class="programlisting">
+#include <boost/spirit/include/lex_lexer_static_lexertl.hpp>
+</pre>
+
+ 
+
+ 
+
+ Otherwise the main program is not different from an equivalent program
+ using the dynamic model. This feature makes it really easy for instance
+ to develop the lexer in dynamic mode and to switch to the static mode after
+ the code has been stabilized. The simple generator application showed above
+ enables the integration of the code generator into any existing build process.
+ The following code snippet provides the overall main function, highlighting
+ the code to be changed.
+ 
+
+ 
+
+
+
+<pre class="programlisting">
+int main(int argc, char* argv[])
+{
+ // Define the token type to be used: 'std::string' is available as the type
+ // of the token value.
+ typedef lexertl_token<
+ char const*, boost::mpl::vector<std::string>
+ > token_type;
+
+ // Define the lexer type to be used as the base class for our token
+ // definition.
+ //
+ // This is the only place where the code is different from an equivalent
+ // dynamic lexical analyzer. We use the `lexertl_static_lexer<>` instead of
+ // the `lexertl_lexer<>` as the base class for our token defintion type.
+ //
+ typedef lexertl_static_lexer<token_type> lexer_type;
+
+ // Define the iterator type exposed by the lexer.
+ typedef lexer_iterator<word_count_tokens<lexer_type> >::type iterator_type;
+
+ // Now we use the types defined above to create the lexer and grammar
+ // object instances needed to invoke the parsing process.
+ word_count_tokens<lexer_type> word_count; // Our token definition
+ word_count_grammar<iterator_type> g (word_count); // Our grammar definition
+
+ // Read in the file into memory.
+ std::string str (read_from_file(1 == argc ? "word_count.input" : argv[1]));
+ char const* first = str.c_str();
+ char const* last = &first[str.size()];
+
+ // Parsing is done based on the the token stream, not the character stream.
+ bool r = tokenize_and_parse(first, last, make_lexer(word_count), g);
+
+ if (r) { // success
+ std::cout << "lines: " << g.l << ", words: " << g.w
+ << ", characters: " << g.c << "\n";
+ }
+ else {
+ std::string rest(first, last);
+ std::cerr << "Parsing failed\n" << "stopped at: \""
+ << rest << "\"\n";
+ }
+ return 0;
+}
+</pre>
+
+ 
+
+ 
+</div>
+<table xmlns:rev="http://www.cs.rpi.edu/~gregod/boost/tools/doc/revision" width="100%"><tr>
+<td align="left"></td>
+<td align="right"><div class="copyright-footer">Copyright © 2001-2008 Joel de Guzman,
+ Hartmut Kaiser
+ Distributed under the Boost Software License, Version 1.0. (See accompanying
+ file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
+ 
+</div></td>
+</tr></table>
+<hr>
+<div class="spirit-nav">
+<a accesskey="p" href="lexer_semantic_actions.html"><img src="../../../../../../../doc/html/images/prev.png" alt="Prev"></a><a accesskey="u" href="../abstracts.html"><img src="../../../../../../../doc/html/images/up.png" alt="Up"></a><a accesskey="h" href="../../../index.html"><img src="../../../../../../../doc/html/images/home.png" alt="Home"></a><a accesskey="n" href="lexer_parsing.html"><img src="../../../../../../../doc/html/images/next.png" alt="Next"></a>
+</div>
+</body>
+</html>

Added: trunk/libs/spirit/doc/html/spirit/lex/abstracts/lexer_tokenizing.html
==============================================================================
--- (empty file)
+++ trunk/libs/spirit/doc/html/spirit/lex/abstracts/lexer_tokenizing.html 2008-07-14 21:20:34 EDT (Mon, 14 Jul 2008)
@@ -0,0 +1,57 @@
+<html>
+<head>
+<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
+<title> Tokenizing
+ Input Data</title>
+<link rel="stylesheet" href="../../../../../../../doc/html/boostbook.css" type="text/css">
+<meta name="generator" content="DocBook XSL Stylesheets V1.66.1">
+<link rel="start" href="../../../index.html" title="Spirit 2.0">
+<link rel="up" href="../abstracts.html" title=" Abstracts">
+<link rel="prev" href="../abstracts.html" title=" Abstracts">
+<link rel="next" href="lexer_semantic_actions.html" title=" Lexer
+ Semantic Actions">
+</head>
+<body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF">
+<table cellpadding="2" width="100%"><tr>
+<td valign="top"><img alt="Boost C++ Libraries" width="277" height="86" src="../../../../../../../boost.png"></td>
+<td align="center">Home</td>
+<td align="center">Libraries</td>
+<td align="center">People</td>
+<td align="center">FAQ</td>
+<td align="center">More</td>
+</tr></table>
+<hr>
+<div class="spirit-nav">
+<a accesskey="p" href="../abstracts.html"><img src="../../../../../../../doc/html/images/prev.png" alt="Prev"></a><a accesskey="u" href="../abstracts.html"><img src="../../../../../../../doc/html/images/up.png" alt="Up"></a><a accesskey="h" href="../../../index.html"><img src="../../../../../../../doc/html/images/home.png" alt="Home"></a><a accesskey="n" href="lexer_semantic_actions.html"><img src="../../../../../../../doc/html/images/next.png" alt="Next"></a>
+</div>
+<div class="section" lang="en">
+<div class="titlepage"><div><div><h4 class="title">
+<a name="spirit.lex.abstracts.lexer_tokenizing"></a><a href="lexer_tokenizing.html" title=" Tokenizing
+ Input Data"> Tokenizing
+ Input Data</a></h4></div></div></div>
+<a name="spirit.lex.abstracts.lexer_tokenizing.the_tokenize___function"></a><h5>
+<a name="id682089"></a>
+ <a href="lexer_tokenizing.html#spirit.lex.abstracts.lexer_tokenizing.the_tokenize___function">The
+ tokenize() function</a>
+ </h5>
+<a name="spirit.lex.abstracts.lexer_tokenizing.the_generate_static___function"></a><h5>
+<a name="id682106"></a>
+ <a href="lexer_tokenizing.html#spirit.lex.abstracts.lexer_tokenizing.the_generate_static___function">The
+ generate_static() function</a>
+ </h5>
+</div>
+<table xmlns:rev="http://www.cs.rpi.edu/~gregod/boost/tools/doc/revision" width="100%"><tr>
+<td align="left"></td>
+<td align="right"><div class="copyright-footer">Copyright © 2001-2008 Joel de Guzman,
+ Hartmut Kaiser
+ Distributed under the Boost Software License, Version 1.0. (See accompanying
+ file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
+ 
+</div></td>
+</tr></table>
+<hr>
+<div class="spirit-nav">
+<a accesskey="p" href="../abstracts.html"><img src="../../../../../../../doc/html/images/prev.png" alt="Prev"></a><a accesskey="u" href="../abstracts.html"><img src="../../../../../../../doc/html/images/up.png" alt="Up"></a><a accesskey="h" href="../../../index.html"><img src="../../../../../../../doc/html/images/home.png" alt="Home"></a><a accesskey="n" href="lexer_semantic_actions.html"><img src="../../../../../../../doc/html/images/next.png" alt="Next"></a>
+</div>
+</body>
+</html>

Added: trunk/libs/spirit/doc/html/spirit/lex/quick_reference.html
==============================================================================
--- (empty file)
+++ trunk/libs/spirit/doc/html/spirit/lex/quick_reference.html 2008-07-14 21:20:34 EDT (Mon, 14 Jul 2008)
@@ -0,0 +1,41 @@
+<html>
+<head>
+<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
+<title> Quick Reference</title>
+<link rel="stylesheet" href="../../../../../../doc/html/boostbook.css" type="text/css">
+<meta name="generator" content="DocBook XSL Stylesheets V1.66.1">
+<link rel="start" href="../../index.html" title="Spirit 2.0">
+<link rel="up" href="../lex.html" title=" Spirit.Lex">
+<link rel="prev" href="abstracts/lexer_states.html" title=" Lexer States">
+<link rel="next" href="reference.html" title=" Reference">
+</head>
+<body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF">
+<table cellpadding="2" width="100%"><tr>
+<td valign="top"><img alt="Boost C++ Libraries" width="277" height="86" src="../../../../../../boost.png"></td>
+<td align="center">Home</td>
+<td align="center">Libraries</td>
+<td align="center">People</td>
+<td align="center">FAQ</td>
+<td align="center">More</td>
+</tr></table>
+<hr>
+<div class="spirit-nav">
+<a accesskey="p" href="abstracts/lexer_states.html"><img src="../../../../../../doc/html/images/prev.png" alt="Prev"></a><a accesskey="u" href="../lex.html"><img src="../../../../../../doc/html/images/up.png" alt="Up"></a><a accesskey="h" href="../../index.html"><img src="../../../../../../doc/html/images/home.png" alt="Home"></a><a accesskey="n" href="reference.html"><img src="../../../../../../doc/html/images/next.png" alt="Next"></a>
+</div>
+<div class="section" lang="en"><div class="titlepage"><div><div><h3 class="title">
+<a name="spirit.lex.quick_reference"></a> Quick Reference</h3></div></div></div></div>
+<table xmlns:rev="http://www.cs.rpi.edu/~gregod/boost/tools/doc/revision" width="100%"><tr>
+<td align="left"></td>
+<td align="right"><div class="copyright-footer">Copyright © 2001-2008 Joel de Guzman,
+ Hartmut Kaiser
+ Distributed under the Boost Software License, Version 1.0. (See accompanying
+ file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
+ 
+</div></td>
+</tr></table>
+<hr>
+<div class="spirit-nav">
+<a accesskey="p" href="abstracts/lexer_states.html"><img src="../../../../../../doc/html/images/prev.png" alt="Prev"></a><a accesskey="u" href="../lex.html"><img src="../../../../../../doc/html/images/up.png" alt="Up"></a><a accesskey="h" href="../../index.html"><img src="../../../../../../doc/html/images/home.png" alt="Home"></a><a accesskey="n" href="reference.html"><img src="../../../../../../doc/html/images/next.png" alt="Next"></a>
+</div>
+</body>
+</html>

Added: trunk/libs/spirit/doc/html/spirit/lex/reference.html
==============================================================================
--- (empty file)
+++ trunk/libs/spirit/doc/html/spirit/lex/reference.html 2008-07-14 21:20:34 EDT (Mon, 14 Jul 2008)
@@ -0,0 +1,68 @@
+<html>
+<head>
+<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
+<title> Reference</title>
+<link rel="stylesheet" href="../../../../../../doc/html/boostbook.css" type="text/css">
+<meta name="generator" content="DocBook XSL Stylesheets V1.66.1">
+<link rel="start" href="../../index.html" title="Spirit 2.0">
+<link rel="up" href="../lex.html" title=" Spirit.Lex">
+<link rel="prev" href="quick_reference.html" title=" Quick Reference">
+<link rel="next" href="reference/lexer_class.html" title="Lexer Class">
+</head>
+<body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF">
+<table cellpadding="2" width="100%"><tr>
+<td valign="top"><img alt="Boost C++ Libraries" width="277" height="86" src="../../../../../../boost.png"></td>
+<td align="center">Home</td>
+<td align="center">Libraries</td>
+<td align="center">People</td>
+<td align="center">FAQ</td>
+<td align="center">More</td>
+</tr></table>
+<hr>
+<div class="spirit-nav">
+<a accesskey="p" href="quick_reference.html"><img src="../../../../../../doc/html/images/prev.png" alt="Prev"></a><a accesskey="u" href="../lex.html"><img src="../../../../../../doc/html/images/up.png" alt="Up"></a><a accesskey="h" href="../../index.html"><img src="../../../../../../doc/html/images/home.png" alt="Home"></a><a accesskey="n" href="reference/lexer_class.html"><img src="../../../../../../doc/html/images/next.png" alt="Next"></a>
+</div>
+<div class="section" lang="en">
+<div class="titlepage"><div><div><h3 class="title">
+<a name="spirit.lex.reference"></a> Reference</h3></div></div></div>
+<div class="toc"><dl>
+<dt> Concepts</dt>
+<dt>Lexer Class</dt>
+<dt>Token Class</dt>
+<dt>TokenDef Class</dt>
+<dt>TokenSet Class</dt>
+</dl></div>
+<div class="section" lang="en">
+<div class="titlepage"><div><div><h4 class="title">
+<a name="spirit.lex.reference.concepts"></a> Concepts</h4></div></div></div>
+<div class="toc"><dl>
+<dt>Lexer</dt>
+<dt>Token</dt>
+<dt>TokenDef</dt>
+<dt>TokenSet</dt>
+</dl></div>
+<div class="section" lang="en"><div class="titlepage"><div><div><h5 class="title">
+<a name="spirit.lex.reference.concepts.lexer"></a>Lexer</h5></div></div></div></div>
+<div class="section" lang="en"><div class="titlepage"><div><div><h5 class="title">
+<a name="spirit.lex.reference.concepts.token"></a>Token</h5></div></div></div></div>
+<div class="section" lang="en"><div class="titlepage"><div><div><h5 class="title">
+<a name="spirit.lex.reference.concepts.tokendef"></a>TokenDef</h5></div></div></div></div>
+<div class="section" lang="en"><div class="titlepage"><div><div><h5 class="title">
+<a name="spirit.lex.reference.concepts.tokenset"></a>TokenSet</h5></div></div></div></div>
+</div>
+</div>
+<table xmlns:rev="http://www.cs.rpi.edu/~gregod/boost/tools/doc/revision" width="100%"><tr>
+<td align="left"></td>
+<td align="right"><div class="copyright-footer">Copyright © 2001-2008 Joel de Guzman,
+ Hartmut Kaiser
+ Distributed under the Boost Software License, Version 1.0. (See accompanying
+ file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
+ 
+</div></td>
+</tr></table>
+<hr>
+<div class="spirit-nav">
+<a accesskey="p" href="quick_reference.html"><img src="../../../../../../doc/html/images/prev.png" alt="Prev"></a><a accesskey="u" href="../lex.html"><img src="../../../../../../doc/html/images/up.png" alt="Up"></a><a accesskey="h" href="../../index.html"><img src="../../../../../../doc/html/images/home.png" alt="Home"></a><a accesskey="n" href="reference/lexer_class.html"><img src="../../../../../../doc/html/images/next.png" alt="Next"></a>
+</div>
+</body>
+</html>

Added: trunk/libs/spirit/doc/html/spirit/lex/reference/lexer_class.html
==============================================================================
--- (empty file)
+++ trunk/libs/spirit/doc/html/spirit/lex/reference/lexer_class.html 2008-07-14 21:20:34 EDT (Mon, 14 Jul 2008)
@@ -0,0 +1,63 @@
+<html>
+<head>
+<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
+<title>Lexer Class</title>
+<link rel="stylesheet" href="../../../../../../../doc/html/boostbook.css" type="text/css">
+<meta name="generator" content="DocBook XSL Stylesheets V1.66.1">
+<link rel="start" href="../../../index.html" title="Spirit 2.0">
+<link rel="up" href="../reference.html" title=" Reference">
+<link rel="prev" href="../reference.html" title=" Reference">
+<link rel="next" href="token_class.html" title="Token Class">
+</head>
+<body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF">
+<table cellpadding="2" width="100%"><tr>
+<td valign="top"><img alt="Boost C++ Libraries" width="277" height="86" src="../../../../../../../boost.png"></td>
+<td align="center">Home</td>
+<td align="center">Libraries</td>
+<td align="center">People</td>
+<td align="center">FAQ</td>
+<td align="center">More</td>
+</tr></table>
+<hr>
+<div class="spirit-nav">
+<a accesskey="p" href="../reference.html"><img src="../../../../../../../doc/html/images/prev.png" alt="Prev"></a><a accesskey="u" href="../reference.html"><img src="../../../../../../../doc/html/images/up.png" alt="Up"></a><a accesskey="h" href="../../../index.html"><img src="../../../../../../../doc/html/images/home.png" alt="Home"></a><a accesskey="n" href="token_class.html"><img src="../../../../../../../doc/html/images/next.png" alt="Next"></a>
+</div>
+<div class="section" lang="en">
+<div class="titlepage"><div><div><h4 class="title">
+<a name="spirit.lex.reference.lexer_class"></a>Lexer Class</h4></div></div></div>
+<a name="spirit.lex.reference.lexer_class.the_lexertl_lexer_class_implementing_the_dynamic_model"></a><h5>
+<a name="id686565"></a>
+ <a href="lexer_class.html#spirit.lex.reference.lexer_class.the_lexertl_lexer_class_implementing_the_dynamic_model">The
+ lexertl_lexer Class Implementing the Dynamic Model</a>
+ </h5>
+<a name="spirit.lex.reference.lexer_class.the_lexertl_actor_lexer_class_implementing_the_dynamic_model"></a><h5>
+<a name="id686578"></a>
+ <a href="lexer_class.html#spirit.lex.reference.lexer_class.the_lexertl_actor_lexer_class_implementing_the_dynamic_model">The
+ lexertl_actor_lexer Class Implementing the Dynamic Model</a>
+ </h5>
+<a name="spirit.lex.reference.lexer_class.the_lexertl_static_lexer_class_implementing_the_static_model"></a><h5>
+<a name="id686591"></a>
+ <a href="lexer_class.html#spirit.lex.reference.lexer_class.the_lexertl_static_lexer_class_implementing_the_static_model">The
+ lexertl_static_lexer Class Implementing the Static Model</a>
+ </h5>
+<a name="spirit.lex.reference.lexer_class.the_lexertl_static_actor_lexer_class_implementing_the_static_model"></a><h5>
+<a name="id686604"></a>
+ <a href="lexer_class.html#spirit.lex.reference.lexer_class.the_lexertl_static_actor_lexer_class_implementing_the_static_model">The
+ lexertl_static_actor_lexer Class Implementing the Static Model</a>
+ </h5>
+</div>
+<table xmlns:rev="http://www.cs.rpi.edu/~gregod/boost/tools/doc/revision" width="100%"><tr>
+<td align="left"></td>
+<td align="right"><div class="copyright-footer">Copyright © 2001-2008 Joel de Guzman,
+ Hartmut Kaiser
+ Distributed under the Boost Software License, Version 1.0. (See accompanying
+ file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
+ 
+</div></td>
+</tr></table>
+<hr>
+<div class="spirit-nav">
+<a accesskey="p" href="../reference.html"><img src="../../../../../../../doc/html/images/prev.png" alt="Prev"></a><a accesskey="u" href="../reference.html"><img src="../../../../../../../doc/html/images/up.png" alt="Up"></a><a accesskey="h" href="../../../index.html"><img src="../../../../../../../doc/html/images/home.png" alt="Home"></a><a accesskey="n" href="token_class.html"><img src="../../../../../../../doc/html/images/next.png" alt="Next"></a>
+</div>
+</body>
+</html>

Added: trunk/libs/spirit/doc/html/spirit/lex/reference/token_class.html
==============================================================================
--- (empty file)
+++ trunk/libs/spirit/doc/html/spirit/lex/reference/token_class.html 2008-07-14 21:20:34 EDT (Mon, 14 Jul 2008)
@@ -0,0 +1,41 @@
+<html>
+<head>
+<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
+<title>Token Class</title>
+<link rel="stylesheet" href="../../../../../../../doc/html/boostbook.css" type="text/css">
+<meta name="generator" content="DocBook XSL Stylesheets V1.66.1">
+<link rel="start" href="../../../index.html" title="Spirit 2.0">
+<link rel="up" href="../reference.html" title=" Reference">
+<link rel="prev" href="lexer_class.html" title="Lexer Class">
+<link rel="next" href="tokendef_class.html" title="TokenDef Class">
+</head>
+<body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF">
+<table cellpadding="2" width="100%"><tr>
+<td valign="top"><img alt="Boost C++ Libraries" width="277" height="86" src="../../../../../../../boost.png"></td>
+<td align="center">Home</td>
+<td align="center">Libraries</td>
+<td align="center">People</td>
+<td align="center">FAQ</td>
+<td align="center">More</td>
+</tr></table>
+<hr>
+<div class="spirit-nav">
+<a accesskey="p" href="lexer_class.html"><img src="../../../../../../../doc/html/images/prev.png" alt="Prev"></a><a accesskey="u" href="../reference.html"><img src="../../../../../../../doc/html/images/up.png" alt="Up"></a><a accesskey="h" href="../../../index.html"><img src="../../../../../../../doc/html/images/home.png" alt="Home"></a><a accesskey="n" href="tokendef_class.html"><img src="../../../../../../../doc/html/images/next.png" alt="Next"></a>
+</div>
+<div class="section" lang="en"><div class="titlepage"><div><div><h4 class="title">
+<a name="spirit.lex.reference.token_class"></a>Token Class</h4></div></div></div></div>
+<table xmlns:rev="http://www.cs.rpi.edu/~gregod/boost/tools/doc/revision" width="100%"><tr>
+<td align="left"></td>
+<td align="right"><div class="copyright-footer">Copyright © 2001-2008 Joel de Guzman,
+ Hartmut Kaiser
+ Distributed under the Boost Software License, Version 1.0. (See accompanying
+ file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
+ 
+</div></td>
+</tr></table>
+<hr>
+<div class="spirit-nav">
+<a accesskey="p" href="lexer_class.html"><img src="../../../../../../../doc/html/images/prev.png" alt="Prev"></a><a accesskey="u" href="../reference.html"><img src="../../../../../../../doc/html/images/up.png" alt="Up"></a><a accesskey="h" href="../../../index.html"><img src="../../../../../../../doc/html/images/home.png" alt="Home"></a><a accesskey="n" href="tokendef_class.html"><img src="../../../../../../../doc/html/images/next.png" alt="Next"></a>
+</div>
+</body>
+</html>

Added: trunk/libs/spirit/doc/html/spirit/lex/reference/tokendef_class.html
==============================================================================
--- (empty file)
+++ trunk/libs/spirit/doc/html/spirit/lex/reference/tokendef_class.html 2008-07-14 21:20:34 EDT (Mon, 14 Jul 2008)
@@ -0,0 +1,41 @@
+<html>
+<head>
+<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
+<title>TokenDef Class</title>
+<link rel="stylesheet" href="../../../../../../../doc/html/boostbook.css" type="text/css">
+<meta name="generator" content="DocBook XSL Stylesheets V1.66.1">
+<link rel="start" href="../../../index.html" title="Spirit 2.0">
+<link rel="up" href="../reference.html" title=" Reference">
+<link rel="prev" href="token_class.html" title="Token Class">
+<link rel="next" href="tokenset_class.html" title="TokenSet Class">
+</head>
+<body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF">
+<table cellpadding="2" width="100%"><tr>
+<td valign="top"><img alt="Boost C++ Libraries" width="277" height="86" src="../../../../../../../boost.png"></td>
+<td align="center">Home</td>
+<td align="center">Libraries</td>
+<td align="center">People</td>
+<td align="center">FAQ</td>
+<td align="center">More</td>
+</tr></table>
+<hr>
+<div class="spirit-nav">
+<a accesskey="p" href="token_class.html"><img src="../../../../../../../doc/html/images/prev.png" alt="Prev"></a><a accesskey="u" href="../reference.html"><img src="../../../../../../../doc/html/images/up.png" alt="Up"></a><a accesskey="h" href="../../../index.html"><img src="../../../../../../../doc/html/images/home.png" alt="Home"></a><a accesskey="n" href="tokenset_class.html"><img src="../../../../../../../doc/html/images/next.png" alt="Next"></a>
+</div>
+<div class="section" lang="en"><div class="titlepage"><div><div><h4 class="title">
+<a name="spirit.lex.reference.tokendef_class"></a>TokenDef Class</h4></div></div></div></div>
+<table xmlns:rev="http://www.cs.rpi.edu/~gregod/boost/tools/doc/revision" width="100%"><tr>
+<td align="left"></td>
+<td align="right"><div class="copyright-footer">Copyright © 2001-2008 Joel de Guzman,
+ Hartmut Kaiser
+ Distributed under the Boost Software License, Version 1.0. (See accompanying
+ file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
+ 
+</div></td>
+</tr></table>
+<hr>
+<div class="spirit-nav">
+<a accesskey="p" href="token_class.html"><img src="../../../../../../../doc/html/images/prev.png" alt="Prev"></a><a accesskey="u" href="../reference.html"><img src="../../../../../../../doc/html/images/up.png" alt="Up"></a><a accesskey="h" href="../../../index.html"><img src="../../../../../../../doc/html/images/home.png" alt="Home"></a><a accesskey="n" href="tokenset_class.html"><img src="../../../../../../../doc/html/images/next.png" alt="Next"></a>
+</div>
+</body>
+</html>

Added: trunk/libs/spirit/doc/html/spirit/lex/reference/tokenset_class.html
==============================================================================
--- (empty file)
+++ trunk/libs/spirit/doc/html/spirit/lex/reference/tokenset_class.html 2008-07-14 21:20:34 EDT (Mon, 14 Jul 2008)
@@ -0,0 +1,41 @@
+<html>
+<head>
+<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
+<title>TokenSet Class</title>
+<link rel="stylesheet" href="../../../../../../../doc/html/boostbook.css" type="text/css">
+<meta name="generator" content="DocBook XSL Stylesheets V1.66.1">
+<link rel="start" href="../../../index.html" title="Spirit 2.0">
+<link rel="up" href="../reference.html" title=" Reference">
+<link rel="prev" href="tokendef_class.html" title="TokenDef Class">
+<link rel="next" href="../../faq.html" title="FAQ">
+</head>
+<body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF">
+<table cellpadding="2" width="100%"><tr>
+<td valign="top"><img alt="Boost C++ Libraries" width="277" height="86" src="../../../../../../../boost.png"></td>
+<td align="center">Home</td>
+<td align="center">Libraries</td>
+<td align="center">People</td>
+<td align="center">FAQ</td>
+<td align="center">More</td>
+</tr></table>
+<hr>
+<div class="spirit-nav">
+<a accesskey="p" href="tokendef_class.html"><img src="../../../../../../../doc/html/images/prev.png" alt="Prev"></a><a accesskey="u" href="../reference.html"><img src="../../../../../../../doc/html/images/up.png" alt="Up"></a><a accesskey="h" href="../../../index.html"><img src="../../../../../../../doc/html/images/home.png" alt="Home"></a><a accesskey="n" href="../../faq.html"><img src="../../../../../../../doc/html/images/next.png" alt="Next"></a>
+</div>
+<div class="section" lang="en"><div class="titlepage"><div><div><h4 class="title">
+<a name="spirit.lex.reference.tokenset_class"></a>TokenSet Class</h4></div></div></div></div>
+<table xmlns:rev="http://www.cs.rpi.edu/~gregod/boost/tools/doc/revision" width="100%"><tr>
+<td align="left"></td>
+<td align="right"><div class="copyright-footer">Copyright © 2001-2008 Joel de Guzman,
+ Hartmut Kaiser
+ Distributed under the Boost Software License, Version 1.0. (See accompanying
+ file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
+ 
+</div></td>
+</tr></table>
+<hr>
+<div class="spirit-nav">
+<a accesskey="p" href="tokendef_class.html"><img src="../../../../../../../doc/html/images/prev.png" alt="Prev"></a><a accesskey="u" href="../reference.html"><img src="../../../../../../../doc/html/images/up.png" alt="Up"></a><a accesskey="h" href="../../../index.html"><img src="../../../../../../../doc/html/images/home.png" alt="Home"></a><a accesskey="n" href="../../faq.html"><img src="../../../../../../../doc/html/images/next.png" alt="Next"></a>
+</div>
+</body>
+</html>

Added: trunk/libs/spirit/doc/html/spirit/lex/tutorials.html
==============================================================================
--- (empty file)
+++ trunk/libs/spirit/doc/html/spirit/lex/tutorials.html 2008-07-14 21:20:34 EDT (Mon, 14 Jul 2008)
@@ -0,0 +1,143 @@
+<html>
+<head>
+<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
+<title> Spirit.Lex
+ Tutorials</title>
+<link rel="stylesheet" href="../../../../../../doc/html/boostbook.css" type="text/css">
+<meta name="generator" content="DocBook XSL Stylesheets V1.66.1">
+<link rel="start" href="../../index.html" title="Spirit 2.0">
+<link rel="up" href="../lex.html" title=" Spirit.Lex">
+<link rel="prev" href="../lex.html" title=" Spirit.Lex">
+<link rel="next" href="tutorials/lexer_quickstart1.html" title=" Quickstart
+ 1 - A word counter using Spirit.Lex">
+</head>
+<body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF">
+<table cellpadding="2" width="100%"><tr>
+<td valign="top"><img alt="Boost C++ Libraries" width="277" height="86" src="../../../../../../boost.png"></td>
+<td align="center">Home</td>
+<td align="center">Libraries</td>
+<td align="center">People</td>
+<td align="center">FAQ</td>
+<td align="center">More</td>
+</tr></table>
+<hr>
+<div class="spirit-nav">
+<a accesskey="p" href="../lex.html"><img src="../../../../../../doc/html/images/prev.png" alt="Prev"></a><a accesskey="u" href="../lex.html"><img src="../../../../../../doc/html/images/up.png" alt="Up"></a><a accesskey="h" href="../../index.html"><img src="../../../../../../doc/html/images/home.png" alt="Home"></a><a accesskey="n" href="tutorials/lexer_quickstart1.html"><img src="../../../../../../doc/html/images/next.png" alt="Next"></a>
+</div>
+<div class="section" lang="en">
+<div class="titlepage"><div><div><h3 class="title">
+<a name="spirit.lex.tutorials"></a><a href="tutorials.html" title=" Spirit.Lex
+ Tutorials"> Spirit.Lex
+ Tutorials</a></h3></div></div></div>
+<div class="toc"><dl>
+<dt><a href="tutorials.html#spirit.lex.tutorials.lexer_tutorials"> Spirit.Lex
+ Tutorials Overview</a></dt>
+<dt><a href="tutorials/lexer_quickstart1.html"> Quickstart
+ 1 - A word counter using Spirit.Lex</a></dt>
+<dt><a href="tutorials/lexer_quickstart2.html"> Quickstart
+ 2 - A better word counter using Spirit.Lex</a></dt>
+<dt><a href="tutorials/lexer_quickstart3.html"> Quickstart
+ 3 - Counting Words Using a Parser</a></dt>
+</dl></div>
+<div class="section" lang="en">
+<div class="titlepage"><div><div><h4 class="title">
+<a name="spirit.lex.tutorials.lexer_tutorials"></a><a href="tutorials.html#spirit.lex.tutorials.lexer_tutorials" title=" Spirit.Lex
+ Tutorials Overview"> Spirit.Lex
+ Tutorials Overview</a></h4></div></div></div>
+
+ The Spirit.Lex library implements several components
+ on top of possibly different lexer generator libraries. It exposes a pair
+ of iterators, which, when dereferenced, return a stream of tokens generated
+ from the underlying character stream. The generated tokens are based on
+ the token definitions supplied by the user.
+ 
+
+ Currently, Spirit.Lex is built on top of Ben Hansons
+ excellent Lexertl
+ library (which is a proposed Boost library). Lexertl
+ provides the necessary functionality to build state machines based on a
+ set of supplied regular expressions. But Spirit.Lex
+ is not restricted to be used with Lexertl.
+ We expect it to be usable in conjunction with any other lexical scanner
+ generator library, all what needs to be implemented is a set of wrapper
+ objects exposing a well defined interface as described in this documentation.
+ 
+<div class="note"><table border="0" summary="Note">
+<tr>
+<td rowspan="2" align="center" valign="top" width="25"><img alt="[Note]" src="../../images/note.png"></td>
+<th align="left">Note</th>
+</tr>
+<tr><td colspan="2" align="left" valign="top">
+ For the sake of clarity all examples in this documentation assume Spirit.Lex
+ to be used on top of Lexertl.
+ </td></tr>
+</table></div>
+
+ Building a lexer using Spirit.Lex is highly configurable,
+ where most of this configuration has to be done at compile time. Almost
+ all of the configurable parameters have generally useful default values,
+ though, which means that starting a project is easy and straightforward.
+ Here is a (non-complete) list of features you can tweak to adjust the generated
+ lexer instance to the actual needs:
+ 
+<div class="itemizedlist"><ul type="disc">
+<li>
+ Select and customize the token type to be generated by the lexer instance.
+ </li>
+<li>
+ Select and customize the token value types the generated token instances
+ will be able to hold.
+ </li>
+<li>
+ Select the iterator type of the underlying input stream, which will be
+ used as the source for the character stream to tokenize.
+ </li>
+<li>
+ Customize the iterator type returned by the lexer to enable debug support,
+ special handling of certain input sequences, etc.
+ </li>
+<li>
+ Select the dynamic or the static
+ runtime model for the lexical analyzer.
+ </li>
+</ul></div>
+
+ Special care has been taken during the development of the library that
+ optimal code will be generated regardless of the configuration options
+ selected.
+ 
+
+ The series of tutorial examples of this section will guide you through
+ some common use cases helping to understand the big picture. The first
+ two quick start examples (<a href="tutorials/lexer_quickstart1.html" title=" Quickstart
+ 1 - A word counter using Spirit.Lex">Lex
+ Quickstart 1 - A word counter using Spirit.Lex</a>
+ and <a href="tutorials/lexer_quickstart2.html" title=" Quickstart
+ 2 - A better word counter using Spirit.Lex">Lex Quickstart
+ 2 - A better word counter using Spirit.Lex</a>)
+ introduce the Spirit.Lex library while building two
+ standalone applications, not being connected to or depending on any other
+ part of Spirit. The
+ section <a href="tutorials/lexer_quickstart3.html" title=" Quickstart
+ 3 - Counting Words Using a Parser">Lex Quickstart
+ 3 - Counting Words Using a Parser</a> demonstrates how to use a lexer
+ in conjunction with a parser (where certainly the parser is built using
+ Spirit.Qi).
+ 
+</div>
+</div>
+<table xmlns:rev="http://www.cs.rpi.edu/~gregod/boost/tools/doc/revision" width="100%"><tr>
+<td align="left"></td>
+<td align="right"><div class="copyright-footer">Copyright © 2001-2008 Joel de Guzman,
+ Hartmut Kaiser
+ Distributed under the Boost Software License, Version 1.0. (See accompanying
+ file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
+ 
+</div></td>
+</tr></table>
+<hr>
+<div class="spirit-nav">
+<a accesskey="p" href="../lex.html"><img src="../../../../../../doc/html/images/prev.png" alt="Prev"></a><a accesskey="u" href="../lex.html"><img src="../../../../../../doc/html/images/up.png" alt="Up"></a><a accesskey="h" href="../../index.html"><img src="../../../../../../doc/html/images/home.png" alt="Home"></a><a accesskey="n" href="tutorials/lexer_quickstart1.html"><img src="../../../../../../doc/html/images/next.png" alt="Next"></a>
+</div>
+</body>
+</html>

Added: trunk/libs/spirit/doc/html/spirit/lex/tutorials/lexer_quickstart1.html
==============================================================================
--- (empty file)
+++ trunk/libs/spirit/doc/html/spirit/lex/tutorials/lexer_quickstart1.html 2008-07-14 21:20:34 EDT (Mon, 14 Jul 2008)
@@ -0,0 +1,362 @@
+<html>
+<head>
+<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
+<title> Quickstart
+ 1 - A word counter using Spirit.Lex</title>
+<link rel="stylesheet" href="../../../../../../../doc/html/boostbook.css" type="text/css">
+<meta name="generator" content="DocBook XSL Stylesheets V1.66.1">
+<link rel="start" href="../../../index.html" title="Spirit 2.0">
+<link rel="up" href="../tutorials.html" title=" Spirit.Lex
+ Tutorials">
+<link rel="prev" href="../tutorials.html" title=" Spirit.Lex
+ Tutorials">
+<link rel="next" href="lexer_quickstart2.html" title=" Quickstart
+ 2 - A better word counter using Spirit.Lex">
+</head>
+<body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF">
+<table cellpadding="2" width="100%"><tr>
+<td valign="top"><img alt="Boost C++ Libraries" width="277" height="86" src="../../../../../../../boost.png"></td>
+<td align="center">Home</td>
+<td align="center">Libraries</td>
+<td align="center">People</td>
+<td align="center">FAQ</td>
+<td align="center">More</td>
+</tr></table>
+<hr>
+<div class="spirit-nav">
+<a accesskey="p" href="../tutorials.html"><img src="../../../../../../../doc/html/images/prev.png" alt="Prev"></a><a accesskey="u" href="../tutorials.html"><img src="../../../../../../../doc/html/images/up.png" alt="Up"></a><a accesskey="h" href="../../../index.html"><img src="../../../../../../../doc/html/images/home.png" alt="Home"></a><a accesskey="n" href="lexer_quickstart2.html"><img src="../../../../../../../doc/html/images/next.png" alt="Next"></a>
+</div>
+<div class="section" lang="en">
+<div class="titlepage"><div><div><h4 class="title">
+<a name="spirit.lex.tutorials.lexer_quickstart1"></a><a href="lexer_quickstart1.html" title=" Quickstart
+ 1 - A word counter using Spirit.Lex"> Quickstart
+ 1 - A word counter using Spirit.Lex</a></h4></div></div></div>
+
+ Spirit.Lex is very modular, which follows the general
+ building principle of the Spirit
+ libraries. You never pay for features you don't use. It is nicely integrated
+ with the other parts of Spirit
+ but nevertheless can be used separately to build standalone lexical analyzers.
+ The first quick start example describes a standalone application: counting
+ characters, words and lines in a file, very similar to what the well known
+ Unix command <tt class="computeroutput">wc</tt> is doing
+ (for the full example code see here: word_count_functor.cpp).
+ 
+<a name="spirit.lex.tutorials.lexer_quickstart1.prerequisites"></a><h5>
+<a name="id670796"></a>
+ Prerequisites
+ </h5>
+
+ The only required <tt class="computeroutput">#include</tt>
+ specific to Spirit.Lex follows. It is a wrapper for
+ all necessary definitions to use Spirit.Lex in a standalone
+ fashion, and on top of the Lexertl
+ library. Additionally we <tt class="computeroutput">#include</tt>
+ two of the Boost headers to define <tt class="computeroutput">boost::bind()</tt> and <tt class="computeroutput">boost::ref()</tt>.
+ 
+
+ 
+
+
+
+<pre class="programlisting">
+#include <boost/spirit/include/lex_lexer_lexertl.hpp>
+#include <boost/bind.hpp>
+#include <boost/ref.hpp>
+</pre>
+
+ 
+
+ 
+
+ To make all the code below more readable we introduce the following namespaces.
+ 
+
+ 
+
+
+
+<pre class="programlisting">
+using namespace boost::spirit;
+using namespace boost::spirit::lex;
+</pre>
+
+ 
+
+ 
+<a name="spirit.lex.tutorials.lexer_quickstart1.defining_tokens"></a><h5>
+<a name="id671491"></a>
+ <a href="lexer_quickstart1.html#spirit.lex.tutorials.lexer_quickstart1.defining_tokens">Defining
+ Tokens</a>
+ </h5>
+
+ The most important step while creating a lexer using Spirit.Lex
+ is to define the tokens to be recognized in the input sequence. This is
+ normally done by defining the regular expressions describing the matching
+ character sequences, and optionally their corresponding token ids. Additionally
+ the defined tokens need to be associated with an instance of a lexer object
+ as provided by the library. The following code snippet shows how this can
+ be done using Spirit.Lex.
+ 
+
+ 
+
+ The template <tt class="computeroutput">word_count_tokens</tt>
+ defines three different tokens: <tt class="computeroutput">ID_WORD</tt>,
+ <tt class="computeroutput">ID_EOL</tt>, and <tt class="computeroutput">ID_CHAR</tt>, representing a word (anything
+ except a whitespace or a newline), a newline character, and any other
+ character (<tt class="computeroutput">ID_WORD</tt>,
+ <tt class="computeroutput">ID_EOL</tt>, and <tt class="computeroutput">ID_CHAR</tt> are enum values representing
+ the token ids, but could be anything else convertible to an integer as
+ well). The direct base class of any token definition class needs to be
+ the template <tt class="computeroutput">lexer_def<></tt>, where the corresponding template
+ parameter (here: <tt class="computeroutput">lexertl_lexer<BaseIterator></tt>) defines which underlying lexer engine
+ has to be used to provide the required state machine functionality. In
+ this example we use the Lexertl based lexer engine as the underlying
+ lexer type.
+ 
+
+ 
+
+
+
+<pre class="programlisting">
+template <typename Lexer>
+struct word_count_tokens : lexer_def<Lexer>
+{
+ template <typename Self>
+ void def (Self& self)
+ {
+ // define tokens (the regular expression to match and the corresponding
+ // token id) and add them to the lexer
+ self.add
+ ("[^ \t\n]+", ID_WORD) // words (anything except ' ', '\t' or '\n')
+ ("\n", ID_EOL) // newline characters
+ (".", ID_CHAR) // anything else is a plain character
+ ;
+ }
+};
+</pre>
+
+ 
+
+ 
+<a name="spirit.lex.tutorials.lexer_quickstart1.doing_the_useful_work"></a><h5>
+<a name="id671838"></a>
+ <a href="lexer_quickstart1.html#spirit.lex.tutorials.lexer_quickstart1.doing_the_useful_work">Doing
+ the Useful Work</a>
+ </h5>
+
+ We will use a setup, where we want the Spirit.Lex
+ library to invoke a given function after any of of the generated tokens
+ is recognized. For this reason we need to implement a functor taking at
+ least the generated token as an argument and returning a boolean value
+ allowing to stop the tokenization process. The default token type used
+ in this example carries a token value of the type <tt class="computeroutput">iterator_range<BaseIterator></tt> pointing to the matched range in the
+ underlying input sequence.
+ 
+
+ 
+
+ In this example the struct 'counter' is used as a functor counting the
+ characters, words and lines in the analyzed input sequence by identifying
+ the matched tokens as passed from the Spirit.Lex
+ library.
+ 
+
+ 
+
+
+
+<pre class="programlisting">
+struct counter
+{
+ // the function operator gets called for each of the matched tokens
+ // c, l, w are references to the counters used to keep track of the numbers
+ template <typename Token>
+ bool operator()(Token const& t, std::size_t& c, std::size_t& w, std::size_t& l) const
+ {
+ switch (t.id()) {
+ case ID_WORD: // matched a word
+ // since we're using a default token type in this example, every
+ // token instance contains a `iterator_range<BaseIterator>` as its
+ // token value pointing to the matched character sequence in the input
+ ++w; c += t.value().size();
+ break;
+ case ID_EOL: // matched a newline character
+ ++l; ++c;
+ break;
+ case ID_CHAR: // matched something else
+ ++c;
+ break;
+ }
+ return true; // always continue to tokenize
+ }
+};
+</pre>
+
+ 
+
+ 
+
+ All what's left is to write some boilerplate code helping to tie together
+ the pieces described so far. To simplify this example we call the <tt class="computeroutput">lex::tokenize()</tt>
+ function implemented in Spirit.Lex (for a more detailed
+ description of this function see here: FIXME),
+ even if we could have written a loop to iterate over the lexer iterators
+ [<tt class="computeroutput">first</tt>, <tt class="computeroutput">last</tt>)
+ as well.
+ 
+<a name="spirit.lex.tutorials.lexer_quickstart1.pulling_everything_together"></a><h5>
+<a name="id672339"></a>
+ <a href="lexer_quickstart1.html#spirit.lex.tutorials.lexer_quickstart1.pulling_everything_together">Pulling
+ Everything Together</a>
+ </h5>
+
+ 
+
+ The main function simply loads the given file into memory (as a <tt class="computeroutput">std::string</tt>), instantiates an instance of
+ the token definition template using the correct iterator type (<tt class="computeroutput">word_count_tokens<char const*></tt>), and finally calls <tt class="computeroutput">lex::tokenize</tt>, passing an instance of the
+ counter functor defined above. The return value of <tt class="computeroutput">lex::tokenize</tt>
+ will be <tt class="computeroutput">true</tt> if the whole
+ input sequence has been successfully tokenized, and <tt class="computeroutput">false</tt>
+ otherwise.
+ 
+
+ 
+
+
+
+<pre class="programlisting">
+int main(int argc, char* argv[])
+{
+ // these variables are used to count characters, words and lines
+ std::size_t c = 0, w = 0, l = 0;
+
+ // read input from the given file
+ std::string str (read_from_file(1 == argc ? "word_count.input" : argv[1]));
+
+ // create the token definition instance needed to invoke the lexical analyzer
+ word_count_tokens<lexertl_lexer<> > word_count_functor;
+
+ // tokenize the given string, the bound functor gets invoked for each of
+ // the matched tokens
+ char const* first = str.c_str();
+ char const* last = &first[str.size()];
+ bool r = lex::tokenize(first, last, make_lexer(word_count_functor),
+ boost::bind(counter(), _1, boost::ref(c), boost::ref(w), boost::ref(l)));
+
+ // print results
+ if (r) {
+ std::cout << "lines: " << l << ", words: " << w
+ << ", characters: " << c << "\n";
+ }
+ else {
+ std::string rest(first, last);
+ std::cout << "Lexical analysis failed\n" << "stopped at: \""
+ << rest << "\"\n";
+ }
+ return 0;
+}
+</pre>
+
+ 
+
+ 
+<a name="spirit.lex.tutorials.lexer_quickstart1.comparing__emphasis_spirit_lex__emphasis__with__ulink_url__http___flex_sourceforge_net___flex__ulink_"></a><h5>
+<a name="id673171"></a>
+ <a href="lexer_quickstart1.html#spirit.lex.tutorials.lexer_quickstart1.comparing__emphasis_spirit_lex__emphasis__with__ulink_url__http___flex_sourceforge_net___flex__ulink_">Comparing
+ Spirit.Lex with Flex</a>
+ </h5>
+
+ This example was deliberately chosen to be similar as much as possible
+ to the equivalent Flex
+ program (see below), which isn't too different from what has to be written
+ when using Spirit.Lex.
+ 
+<div class="note"><table border="0" summary="Note">
+<tr>
+<td rowspan="2" align="center" valign="top" width="25"><img alt="[Note]" src="../../../images/note.png"></td>
+<th align="left">Note</th>
+</tr>
+<tr><td colspan="2" align="left" valign="top">
+ Interestingly enough, performance comparisons of lexical analyzers written
+ using Spirit.Lex with equivalent programs generated
+ by Flex show that both
+ have comparable execution speeds! Generally, thanks to the highly optimized
+ Lexertl library
+ and due its carefully designed integration with Spirit
+ the abstraction penalty to be paid for using Spirit.Lex
+ is neglectible.
+ </td></tr>
+</table></div>
+
+ The remaining examples in this tutorial will use more sophisticated features
+ of Spirit.Lex, mainly to allow further simplification
+ of the code to be written, while maintaining the similarity with corresponding
+ features of Flex. Spirit.Lex
+ has been designed to be as much as possible similar to Flex,
+ that is why this documentation will provide the corresponding Flex
+ code for the shown Spirit.Lex examples almost everywhere.
+ So consequently, here is the Flex
+ code corresponding to the example as shown above.
+ 
+
+ 
+
+
+
+<pre class="programlisting">
+%{
+ #define ID_WORD 1000
+ #define ID_EOL 1001
+ #define ID_CHAR 1002
+ int c = 0, w = 0, l = 0;
+%}
+%%
+[^ \t\n]+ { return ID_WORD; }
+\n { return ID_EOL; }
+. { return ID_CHAR; }
+%%
+bool count(int tok)
+{
+ switch (tok) {
+ case ID_WORD: ++w; c += yyleng; break;
+ case ID_EOL: ++l; ++c; break;
+ case ID_CHAR: ++c; break;
+ default:
+ return false;
+ }
+ return true;
+}
+void main()
+{
+ int tok = EOF;
+ do {
+ tok = yylex();
+ if (!count(tok))
+ break;
+ } while (EOF != tok);
+ printf("%d %d %d\n", l, w, c);
+}
+</pre>
+
+ 
+
+ 
+</div>
+<table xmlns:rev="http://www.cs.rpi.edu/~gregod/boost/tools/doc/revision" width="100%"><tr>
+<td align="left"></td>
+<td align="right"><div class="copyright-footer">Copyright © 2001-2008 Joel de Guzman,
+ Hartmut Kaiser
+ Distributed under the Boost Software License, Version 1.0. (See accompanying
+ file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
+ 
+</div></td>
+</tr></table>
+<hr>
+<div class="spirit-nav">
+<a accesskey="p" href="../tutorials.html"><img src="../../../../../../../doc/html/images/prev.png" alt="Prev"></a><a accesskey="u" href="../tutorials.html"><img src="../../../../../../../doc/html/images/up.png" alt="Up"></a><a accesskey="h" href="../../../index.html"><img src="../../../../../../../doc/html/images/home.png" alt="Home"></a><a accesskey="n" href="lexer_quickstart2.html"><img src="../../../../../../../doc/html/images/next.png" alt="Next"></a>
+</div>
+</body>
+</html>

Added: trunk/libs/spirit/doc/html/spirit/lex/tutorials/lexer_quickstart2.html
==============================================================================
--- (empty file)
+++ trunk/libs/spirit/doc/html/spirit/lex/tutorials/lexer_quickstart2.html 2008-07-14 21:20:34 EDT (Mon, 14 Jul 2008)
@@ -0,0 +1,295 @@
+<html>
+<head>
+<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
+<title> Quickstart
+ 2 - A better word counter using Spirit.Lex</title>
+<link rel="stylesheet" href="../../../../../../../doc/html/boostbook.css" type="text/css">
+<meta name="generator" content="DocBook XSL Stylesheets V1.66.1">
+<link rel="start" href="../../../index.html" title="Spirit 2.0">
+<link rel="up" href="../tutorials.html" title=" Spirit.Lex
+ Tutorials">
+<link rel="prev" href="lexer_quickstart1.html" title=" Quickstart
+ 1 - A word counter using Spirit.Lex">
+<link rel="next" href="lexer_quickstart3.html" title=" Quickstart
+ 3 - Counting Words Using a Parser">
+</head>
+<body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF">
+<table cellpadding="2" width="100%"><tr>
+<td valign="top"><img alt="Boost C++ Libraries" width="277" height="86" src="../../../../../../../boost.png"></td>
+<td align="center">Home</td>
+<td align="center">Libraries</td>
+<td align="center">People</td>
+<td align="center">FAQ</td>
+<td align="center">More</td>
+</tr></table>
+<hr>
+<div class="spirit-nav">
+<a accesskey="p" href="lexer_quickstart1.html"><img src="../../../../../../../doc/html/images/prev.png" alt="Prev"></a><a accesskey="u" href="../tutorials.html"><img src="../../../../../../../doc/html/images/up.png" alt="Up"></a><a accesskey="h" href="../../../index.html"><img src="../../../../../../../doc/html/images/home.png" alt="Home"></a><a accesskey="n" href="lexer_quickstart3.html"><img src="../../../../../../../doc/html/images/next.png" alt="Next"></a>
+</div>
+<div class="section" lang="en">
+<div class="titlepage"><div><div><h4 class="title">
+<a name="spirit.lex.tutorials.lexer_quickstart2"></a><a href="lexer_quickstart2.html" title=" Quickstart
+ 2 - A better word counter using Spirit.Lex"> Quickstart
+ 2 - A better word counter using Spirit.Lex</a></h4></div></div></div>
+
+ People knowing Flex will
+ probably complain about the example from the section <a href="lexer_quickstart1.html" title=" Quickstart
+ 1 - A word counter using Spirit.Lex">Lex
+ Quickstart 1 - A word counter using Spirit.Lex</a>
+ as being overly complex and not being written to leverage the possibilities
+ provided by this tool. In particular the previous example did not directly
+ use the lexer actions to count the lines, words and characters. So the
+ example provided in this step of the tutorial will show how to use semantic
+ actions in Spirit.Lex. Even if it still will allow
+ to count text elements only it introduces other new concepts and configuration
+ options along the lines (for the full example code see here: word_count_lexer.cpp).
+ 
+<a name="spirit.lex.tutorials.lexer_quickstart2.prerequisites"></a><h5>
+<a name="id673962"></a>
+ Prerequisites
+ </h5>
+
+ In addition to the only required <tt class="computeroutput">#include</tt>
+ specific to Spirit.Lex this example needs to include
+ a couple of header files from the Phoenix2
+ library. This example shows how to attach functors to token definitions,
+ which could be done using any type of C++ technique resulting in a callable
+ object. Using Phoenix2
+ for this task simplifies things and avoids adding dependencies to other
+ libraries (Phoenix2
+ is already in use for Spirit
+ anyway).
+ 
+
+ 
+
+
+
+<pre class="programlisting">
+#include <boost/spirit/include/support_argument.hpp>
+#include <boost/spirit/include/lex_lexer_lexertl.hpp>
+#include <boost/spirit/include/phoenix_operator.hpp>
+#include <boost/spirit/include/phoenix_statement.hpp>
+#include <boost/spirit/include/phoenix_algorithm.hpp>
+#include <boost/spirit/include/phoenix_core.hpp>
+</pre>
+
+ 
+
+ 
+
+ To make all the code below more readable we introduce the following namespaces.
+ 
+
+ 
+
+
+
+<pre class="programlisting">
+using namespace boost::spirit;
+using namespace boost::spirit::lex;
+</pre>
+
+ 
+
+ 
+
+ To give a preview at what to expect from this example, here is the flex
+ program which has been used as the starting point. The useful code is directly
+ included inside the actions associated with each of the token definitions.
+ 
+
+ 
+
+
+
+<pre class="programlisting">
+%{
+ int c = 0, w = 0, l = 0;
+%}
+%%
+[^ \t\n]+ { ++w; c += yyleng; }
+\n { ++c; ++l; }
+. { ++c; }
+%%
+main()
+{
+ yylex();
+ printf("%d %d %d\n", l, w, c);
+}
+</pre>
+
+ 
+
+ 
+<a name="spirit.lex.tutorials.lexer_quickstart2.semantic_actions_in__emphasis_spirit_lex__emphasis_"></a><h5>
+<a name="id674661"></a>
+ <a href="lexer_quickstart2.html#spirit.lex.tutorials.lexer_quickstart2.semantic_actions_in__emphasis_spirit_lex__emphasis_">Semantic
+ Actions in Spirit.Lex</a>
+ </h5>
+
+ Spirit.Lex uses a very similar way of associating
+ actions with the token definitions (which should look familiar to anybody
+ knowlegdeable with Spirit
+ as well): specifying the operations to execute inside of a pair of <tt class="computeroutput">[]</tt> brackets. In order to be able to attach
+ semantic actions to token definitions for each of them there is defined
+ an instance of a <tt class="computeroutput">token_def<></tt>.
+ 
+
+ 
+
+
+
+<pre class="programlisting">
+template <typename Lexer>
+struct word_count_tokens : lexer_def<Lexer>
+{
+ word_count_tokens()
+ : c(0), w(0), l(0),
+ word("[^ \t\n]+"), eol("\n"), any(".") // define tokens
+ {}
+
+ template <typename Self>
+ void def (Self& self)
+ {
+ using boost::phoenix::ref;
+ using boost::phoenix::distance;
+ using boost::spirit::arg_names::_1;
+
+ // associate tokens with the lexer
+ self = word [++ref(w), ref(c) += distance(_1)]
+ | eol [++ref(c), ++ref(l)]
+ | any [++ref(c)]
+ ;
+ }
+
+ std::size_t c, w, l;
+ token_def<> word, eol, any;
+};
+</pre>
+
+ 
+
+ 
+
+ The semantics of the shown code is as follows. The code inside the <tt class="computeroutput">[]</tt> brackets will be executed whenever the
+ corresponding token has been matched by the lexical analyzer. This is very
+ similar to Flex, where
+ the action code associated with a token definition gets executed after
+ the recognition of a matching input sequence. The code above uses function
+ objects constructed using Phoenix2,
+ but it is possible to insert any C++ function or function object as long
+ as it exposes the interface:
+ 
+<pre class="programlisting">
+void f (Range r, Idtype id, bool& matched, Context& ctx);
+</pre>
+<div class="variablelist">
+where:
+<dl>
+<dt><tt class="computeroutput">Range
+ r</tt></dt>
+<dd>
+ This is a <tt class="computeroutput">boost::iterator_range</tt> holding two iterators
+ pointing to the matched range in the underlying input sequence. The
+ type of the held iterators is the same as specified while defining
+ the type of the <tt class="computeroutput">lexertl_lexer<...></tt> (its first template parameter).
+ </dd>
+<dt><tt class="computeroutput">Idtype
+ id</tt></dt>
+<dd>
+ This is the token id of the type <tt class="computeroutput">std::size_t</tt>
+ for the matched token.
+ </dd>
+<dt><tt class="computeroutput">bool& matched</tt></dt>
+<dd>
+ This boolean value is pre/initialized to <tt class="computeroutput">true</tt>.
+ If the functor sets it to <tt class="computeroutput">false</tt>
+ the lexer stops calling any semantic actions attached to this token
+ and behaves as if the token have not been matched in the first place.
+ </dd>
+<dt><tt class="computeroutput">Context& ctx</tt></dt>
+<dd>
+ This is a reference to a lexer specific, unspecified type, providing
+ the context for the current lexer state. It can be used to access different
+ internal data items and is needed for lexer state control from inside
+ a semantic action.
+ </dd>
+</dl>
+</div>
+
+ When using a C++ function as the semantic action the following prototypes
+ are allowed as well:
+ 
+<pre class="programlisting">
+void f (Range r, Idtype id, bool& matched);
+void f (Range r, Idtype id);
+void f (Range r);
+</pre>
+
+ Even if it is possible to write your own function object implementations
+ (i.e. using Boost.Lambda or Boost.Bind), the preferred way of defining
+ lexer semantic actions is to use Phoenix2.
+ In this case you can access the four parameters described in the table
+ above by using the predefined Spirit
+ placeholders: <tt class="computeroutput">_1</tt> for the
+ iterator range, <tt class="computeroutput">_2</tt> for
+ the token id, <tt class="computeroutput">_3</tt> for the
+ reference to the boolean value signaling the outcome of the semantic action,
+ and <tt class="computeroutput">_4</tt> for the reference
+ to the internal lexer context.
+ 
+<a name="spirit.lex.tutorials.lexer_quickstart2.associating_token_definitions_with_the_lexer"></a><h5>
+<a name="id675706"></a>
+ <a href="lexer_quickstart2.html#spirit.lex.tutorials.lexer_quickstart2.associating_token_definitions_with_the_lexer">Associating
+ Token Definitions with the Lexer</a>
+ </h5>
+
+ If you compare the with the code from <a href="lexer_quickstart1.html" title=" Quickstart
+ 1 - A word counter using Spirit.Lex">Lex
+ Quickstart 1 - A word counter using Spirit.Lex</a>
+ with regard to the way how token definitions are associated with the lexer,
+ you will notice a different syntax being used here. If in the previous
+ example we have been using the <tt class="computeroutput">self.add()</tt> style of the API, then here we directly
+ assign the token definitions to <tt class="computeroutput">self</tt>,
+ combining the different token definitions using the <tt class="computeroutput">|</tt>
+ operator. Here is the code snippet again:
+ 
+<pre class="programlisting">
+self = word [++ref(w), ref(c) += distance(_1)]
+ | eol [++ref(c), ++ref(l)]
+ | any [++ref(c)]
+ ;
+</pre>
+
+ This way we have a very powerful and natural way of building the lexical
+ analyzer. If translated into English this may be read as: The lexical analyer
+ will recognize ('<tt class="computeroutput">=</tt>') tokens
+ as defined by any of ('<tt class="computeroutput">|</tt>')
+ the token definitions <tt class="computeroutput">word</tt>,
+ <tt class="computeroutput">eol</tt>, and <tt class="computeroutput">any</tt>.
+ 
+
+ A second difference to the previous example is that we do not explicitly
+ specify any token ids to use for the separate tokens. Using semantic actions
+ to trigger some useful work free'd us from the need to define these. To
+ ensure every token gets assigned a id the Spirit.Lex
+ library internally assigns unique numbers to the token definitions, starting
+ with the constant defined by <tt class="computeroutput">boost::spirit::lex::min_token_id</tt>.
+ 
+</div>
+<table xmlns:rev="http://www.cs.rpi.edu/~gregod/boost/tools/doc/revision" width="100%"><tr>
+<td align="left"></td>
+<td align="right"><div class="copyright-footer">Copyright © 2001-2008 Joel de Guzman,
+ Hartmut Kaiser
+ Distributed under the Boost Software License, Version 1.0. (See accompanying
+ file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
+ 
+</div></td>
+</tr></table>
+<hr>
+<div class="spirit-nav">
+<a accesskey="p" href="lexer_quickstart1.html"><img src="../../../../../../../doc/html/images/prev.png" alt="Prev"></a><a accesskey="u" href="../tutorials.html"><img src="../../../../../../../doc/html/images/up.png" alt="Up"></a><a accesskey="h" href="../../../index.html"><img src="../../../../../../../doc/html/images/home.png" alt="Home"></a><a accesskey="n" href="lexer_quickstart3.html"><img src="../../../../../../../doc/html/images/next.png" alt="Next"></a>
+</div>
+</body>
+</html>

Added: trunk/libs/spirit/doc/html/spirit/lex/tutorials/lexer_quickstart3.html
==============================================================================
--- (empty file)
+++ trunk/libs/spirit/doc/html/spirit/lex/tutorials/lexer_quickstart3.html 2008-07-14 21:20:34 EDT (Mon, 14 Jul 2008)
@@ -0,0 +1,460 @@
+<html>
+<head>
+<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
+<title> Quickstart
+ 3 - Counting Words Using a Parser</title>
+<link rel="stylesheet" href="../../../../../../../doc/html/boostbook.css" type="text/css">
+<meta name="generator" content="DocBook XSL Stylesheets V1.66.1">
+<link rel="start" href="../../../index.html" title="Spirit 2.0">
+<link rel="up" href="../tutorials.html" title=" Spirit.Lex
+ Tutorials">
+<link rel="prev" href="lexer_quickstart2.html" title=" Quickstart
+ 2 - A better word counter using Spirit.Lex">
+<link rel="next" href="../abstracts.html" title=" Abstracts">
+</head>
+<body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF">
+<table cellpadding="2" width="100%"><tr>
+<td valign="top"><img alt="Boost C++ Libraries" width="277" height="86" src="../../../../../../../boost.png"></td>
+<td align="center">Home</td>
+<td align="center">Libraries</td>
+<td align="center">People</td>
+<td align="center">FAQ</td>
+<td align="center">More</td>
+</tr></table>
+<hr>
+<div class="spirit-nav">
+<a accesskey="p" href="lexer_quickstart2.html"><img src="../../../../../../../doc/html/images/prev.png" alt="Prev"></a><a accesskey="u" href="../tutorials.html"><img src="../../../../../../../doc/html/images/up.png" alt="Up"></a><a accesskey="h" href="../../../index.html"><img src="../../../../../../../doc/html/images/home.png" alt="Home"></a><a accesskey="n" href="../abstracts.html"><img src="../../../../../../../doc/html/images/next.png" alt="Next"></a>
+</div>
+<div class="section" lang="en">
+<div class="titlepage"><div><div><h4 class="title">
+<a name="spirit.lex.tutorials.lexer_quickstart3"></a><a href="lexer_quickstart3.html" title=" Quickstart
+ 3 - Counting Words Using a Parser"> Quickstart
+ 3 - Counting Words Using a Parser</a></h4></div></div></div>
+
+ The whole purpose of integrating Spirit.Lex as part
+ of the Spirit library
+ was to add a library allowing to merge lexical analysis with the parsing
+ process as defined by a Spirit
+ grammar. Spirit parsers
+ read their input from an input sequence accessed by iterators. So naturally,
+ we chose iterators to be used as the interface beween the lexer and the
+ parser. A second goal of the lexer/parser integration was to enable the
+ usage of possibly different lexical analyzer libraries. The utilization
+ of iterators seemed to be the right choice from this standpoint as well,
+ mainly because these can be used as an abstraction layer hiding implementation
+ specifics of the used lexer library. The <a href="lexer_quickstart3.html#spirit.lex.flowcontrol" title="Figure 4. The common flow control implemented while parsing
+ combined with lexical analysis">picture</a>
+ below shows the common flow control implemented while parsing combined
+ with lexical analysis.
+ 
+
+ 
+<div class="figure">
+<a name="spirit.lex.flowcontrol"></a>Figure 4. The common flow control implemented while parsing
+ combined with lexical analysis
+<img src="../../.././images/FlowOfControl.png" alt="The common flow
+ control implemented while parsing combined with lexical analysis">
+</div>
+
+ 
+
+ Another problem related to the integration of the lexical analyzer with
+ the parser was to find a way how the defined tokens syntactically could
+ be blended with the grammar definition syntax of Spirit.
+ For tokens defined as instances of the <tt class="computeroutput">token_def<></tt> class the most natural way of integration
+ was to allow to directly use these as parser components. Semantically these
+ parser components succeed matching their input whenever the corresponding
+ token type has been matched by the lexer. This quick start example will
+ demonstrate this (and more) by counting words again, simply by adding up
+ the numbers inside of semantic actions of a parser (for the full example
+ code see here: word_count.cpp).
+ 
+<a name="spirit.lex.tutorials.lexer_quickstart3.prerequisites"></a><h5>
+<a name="id677329"></a>
+ Prerequisites
+ </h5>
+
+ This example uses two of the Spirit
+ library components: Spirit.Lex and Spirit.Qi,
+ consequently we have to <tt class="computeroutput">#include</tt>
+ the corresponding header files. Again, we need to include a couple of header
+ files from the Phoenix2
+ library. This example shows how to attach functors to parser components,
+ which could be done using any type of C++ technique resulting in a callable
+ object. Using Phoenix2
+ for this task simplifies things and avoids adding dependencies to other
+ libraries (Phoenix2
+ is already in use for Spirit
+ anyway).
+ 
+
+ 
+
+
+
+<pre class="programlisting">
+#include <boost/spirit/include/qi.hpp>
+#include <boost/spirit/include/lex_lexer_lexertl.hpp>
+#include <boost/spirit/include/phoenix_operator.hpp>
+#include <boost/spirit/include/phoenix_statement.hpp>
+#include <boost/spirit/include/phoenix_container.hpp>
+</pre>
+
+ 
+
+ 
+
+ To make all the code below more readable we introduce the following namespaces.
+ 
+
+ 
+
+
+
+<pre class="programlisting">
+using namespace boost::spirit;
+using namespace boost::spirit::qi;
+using namespace boost::spirit::lex;
+</pre>
+
+ 
+
+ 
+<a name="spirit.lex.tutorials.lexer_quickstart3.defining_tokens"></a><h5>
+<a name="id677730"></a>
+ <a href="lexer_quickstart3.html#spirit.lex.tutorials.lexer_quickstart3.defining_tokens">Defining
+ Tokens</a>
+ </h5>
+
+ If compared to the two previous quick start examples (<a href="lexer_quickstart1.html" title=" Quickstart
+ 1 - A word counter using Spirit.Lex">Lex
+ Quickstart 1 - A word counter using Spirit.Lex</a>
+ and <a href="lexer_quickstart2.html" title=" Quickstart
+ 2 - A better word counter using Spirit.Lex">Lex Quickstart
+ 2 - A better word counter using Spirit.Lex</a>)
+ the token definition class for this example does not reveal any surprises.
+ However, it uses lexer token definition macros to simplify the composition
+ of the regular expressions, which will be described in more detail in the
+ section FIXME. Generally, any token definition
+ is usable without modification either for a standalone lexical analyzer
+ or in conjunction with a parser.
+ 
+
+ 
+
+
+
+<pre class="programlisting">
+template <typename Lexer>
+struct word_count_tokens : lexer_def<Lexer>
+{
+ template <typename Self>
+ void def (Self& self)
+ {
+ // define patterns (lexer macros) to be used during token definition
+ // below
+ self.add_pattern
+ ("WORD", "[^ \t\n]+")
+ ;
+
+ // define tokens and associate them with the lexer
+ word = "{WORD}"; // reference the pattern 'WORD' as defined above
+
+ // this lexer will recognize 3 token types: words, newlines, and
+ // everything else
+ self.add
+ (word) // no token id is needed here
+ ('\n') // characters are usable as tokens as well
+ (".", IDANY)
+ ;
+ }
+
+ token_def<std::string> word;
+};
+</pre>
+
+ 
+
+ 
+<a name="spirit.lex.tutorials.lexer_quickstart3.using_token_definition_instances_as_parsers"></a><h5>
+<a name="id678087"></a>
+ <a href="lexer_quickstart3.html#spirit.lex.tutorials.lexer_quickstart3.using_token_definition_instances_as_parsers">Using
+ Token Definition Instances as Parsers</a>
+ </h5>
+
+ While the integration of lexer and parser in the control flow is achieved
+ by using special iterators wrapping the lexical analyzer, we still nead
+ a means of expressing in the grammar what tokens to match and where. The
+ token definition class above uses three different ways of defining a token:
+ 
+<div class="itemizedlist"><ul type="disc">
+<li>
+ Using an instance of a <tt class="computeroutput">token_def<></tt>, which is handy whenever you
+ need to specify a token attribute (for more information about lexer related
+ attributes please look here: <a href="../abstracts/lexer_attributes.html" title=" Lexer Attributes">Lexer
+ Attributes</a>).
+ </li>
+<li>
+ Using a single character as the token, in this case the character represents
+ itself as a token, where the token id is the ASCII character value.
+ </li>
+<li>
+ Using a regular expression represented as a string, where the token id
+ needs to be specified explicitly to make the token accessible from the
+ grammar level.
+ </li>
+</ul></div>
+
+ All three token definition methods require a different method of grammar
+ integration. But as you can see from the following code snippet, each of
+ this methods is straightforward and blends the corresponding token instance
+ naturally with the surrounding Spirit.Qi grammar syntax.
+ 
+<div class="informaltable"><table class="table">
+<colgroup>
+<col>
+<col>
+</colgroup>
+<thead><tr>
+<th>
+ 
+ Token definition
+ 
+ </th>
+<th>
+ 
+ Parser integration
+ 
+ </th>
+</tr></thead>
+<tbody>
+<tr>
+<td>
+ 
+ <tt class="computeroutput">token_def<></tt>
+ 
+ </td>
+<td>
+ 
+ The <tt class="computeroutput">token_def<></tt>
+ instance is directly usable as a parser component. Parsing of this
+ component will succeed if the regular expression used to define
+ this has been matched successfully.
+ 
+ </td>
+</tr>
+<tr>
+<td>
+ 
+ single character
+ 
+ </td>
+<td>
+ 
+ The single character is directly usable in the grammar, under certain
+ circumstances it needs to be wrapped by a <tt class="computeroutput">char_()</tt> parser component, though. Parsing
+ of this component will succeed if the single character has been
+ matched.
+ 
+ </td>
+</tr>
+<tr>
+<td>
+ 
+ explicit token id
+ 
+ </td>
+<td>
+ 
+ To use an explicit token id in a Spirit.Qi
+ grammar you are required to wrap it with the special <tt class="computeroutput">token()</tt>
+ parser component. Parsing of this component will succeed if the
+ current token has the same token id as specified in the expression
+ <tt class="computeroutput">token(<id>)</tt>.
+ 
+ </td>
+</tr>
+</tbody>
+</table></div>
+
+ The grammar definition below uses each of the three types demonstrating
+ their usage.
+ 
+
+ 
+
+
+
+<pre class="programlisting">
+template <typename Iterator>
+struct word_count_grammar : grammar<Iterator>
+{
+ template <typename TokenDef>
+ word_count_grammar(TokenDef const& tok)
+ : grammar<Iterator>(start), c(0), w(0), l(0)
+ {
+ using boost::phoenix::ref;
+ using boost::phoenix::size;
+
+ // As documented in the Spirit.Qi documentation, any placeholders
+ // (_1 et.al.) used in semantic actions inside a grammar need to be
+ // imported from the namespace boost::spirit::arg_names, and not from
+ // the corresponding namespace in Phoenix.
+ using boost::spirit::arg_names::_1;
+
+ start = *( tok.word [++ref(w), ref(c) += size(_1)]
+ | char_('\n') [++ref(c), ++ref(l)]
+ | token(IDANY) [++ref(c)]
+ )
+ ;
+ }
+
+ std::size_t c, w, l;
+ rule<Iterator> start;
+};
+</pre>
+
+ 
+
+ 
+
+ As already described (see: Attributes),
+ the Spirit.Qi parser library builds upon a set of
+ of fully attributed parser components. Consequently, all the token definitions
+ do support the this attribute model as well. The most natural way of implementing
+ this was to use the token values as the attributes exposed by the parser
+ component corresponding to the token definition (you can read more about
+ this topic here: <a href="../abstracts.html#spirit.lex.abstracts.lexer_primitives.lexer_token_values" title="
+ About Tokens and Token Values">About
+ Tokens and Token Values</a>). The example above takes advantage of the
+ full integration of the token values as the <tt class="computeroutput">token_def<></tt>'s parser attributes: the <tt class="computeroutput">word</tt> token definition is declared as
+ a <tt class="computeroutput">token_def<std::string></tt>,
+ making every instance of a <tt class="computeroutput">word</tt>
+ token carry the string representation of the matched input sequence as
+ its value. The semantic action attached to <tt class="computeroutput">tok.word</tt>
+ receives this string (represented by the <tt class="computeroutput">_1</tt>
+ placeholder) and uses it to calculate the number of matched characters:
+ <tt class="computeroutput">ref(c) +=
+ size(_1)</tt>.
+ 
+<div class="important"><table border="0" summary="Important">
+<tr>
+<td rowspan="2" align="center" valign="top" width="25"><img alt="[Important]" src="../../../images/important.png"></td>
+<th align="left">Important</th>
+</tr>
+<tr><td colspan="2" align="left" valign="top">
+ All placeholders (<tt class="computeroutput">_1</tt>,
+ <tt class="computeroutput">_2</tt>, etc.) used in parser
+ semantic actions in conjunction with functors created based on Phoenix2 need to be imported
+ from the <tt class="computeroutput">namespace boost::spirit::arg_names</tt>
+ (and not <tt class="computeroutput">namespace
+ boost::phoenix::arg_names</tt>, which is different from
+ using placeholders in Spirit.Lex). Using the wrong
+ placeholders leads to subtle compilation errors which are difficult to
+ backtrack to their cause.
+ </td></tr>
+</table></div>
+<a name="spirit.lex.tutorials.lexer_quickstart3.pulling_everything_together"></a><h5>
+<a name="id679085"></a>
+ <a href="lexer_quickstart3.html#spirit.lex.tutorials.lexer_quickstart3.pulling_everything_together">Pulling
+ Everything Together</a>
+ </h5>
+
+ The main function needs to implement a bit more logic now as we have to
+ initialize and start not only the lexical analysis but the parsing process
+ as well. The three type definitions (<tt class="computeroutput">typedef</tt>
+ statements) simplify the creation of the lexical analyzer and the grammar.
+ After reading the contents of the given file into memory it calls the function
+ tokenize_and_parse()
+ to initialize the lexical analysis and parsing processes.
+ 
+
+ 
+
+
+
+<pre class="programlisting">
+int main(int argc, char* argv[])
+{
+<a name="spirit4co" href="lexer_quickstart3.html#spirit4"><img src="../../../images/callouts/1.png" alt="1" border="0"></a> typedef lexertl_token<
+ char const*, boost::mpl::vector<std::string>
+ > token_type;
+
+<a name="spirit5co" href="lexer_quickstart3.html#spirit5"><img src="../../../images/callouts/2.png" alt="2" border="0"></a> typedef lexertl_lexer<token_type> lexer_type;
+
+<a name="spirit6co" href="lexer_quickstart3.html#spirit6"><img src="../../../images/callouts/3.png" alt="3" border="0"></a> typedef lexer_iterator<word_count_tokens<lexer_type> >::type iterator_type;
+
+ // now we use the types defined above to create the lexer and grammar
+ // object instances needed to invoke the parsing process
+ word_count_tokens<lexer_type> word_count; // Our token definition
+ word_count_grammar<iterator_type> g (word_count); // Our grammar definition
+
+ // read in the file int memory
+ std::string str (read_from_file(1 == argc ? "word_count.input" : argv[1]));
+ char const* first = str.c_str();
+ char const* last = &first[str.size()];
+
+ // Parsing is done based on the the token stream, not the character
+ // stream read from the input. The function `tokenize_and_parse()` wraps
+ // the passed iterator range `[first, last)` by the lexical analyzer and
+ // uses its exposed iterators to parse the toke stream.
+ bool r = tokenize_and_parse(first, last, make_lexer(word_count), g);
+
+ if (r) {
+ std::cout << "lines: " << g.l << ", words: " << g.w
+ << ", characters: " << g.c << "\n";
+ }
+ else {
+ std::string rest(first, last);
+ std::cerr << "Parsing failed\n" << "stopped at: \""
+ << rest << "\"\n";
+ }
+ return 0;
+}
+</pre>
+
+ 
+
+ 
+
+ 
+<div class="calloutlist"><table border="0" summary="Callout list">
+<tr>
+<td width="5%" valign="top" align="left">
+<a name="spirit4"></a> </td>
+<td valign="top" align="left"> define the token type to be used: <tt class="computeroutput">std::string</tt>
+ is available as the type of the token value </td>
+</tr>
+<tr>
+<td width="5%" valign="top" align="left">
+<a name="spirit5"></a> </td>
+<td valign="top" align="left"> define the lexer type to
+ use implementing the state machine </td>
+</tr>
+<tr>
+<td width="5%" valign="top" align="left">
+<a name="spirit6"></a> </td>
+<td valign="top" align="left"> define the iterator type exposed by
+ the lexer type </td>
+</tr>
+</table></div>
+
+ 
+
+ 
+</div>
+<table xmlns:rev="http://www.cs.rpi.edu/~gregod/boost/tools/doc/revision" width="100%"><tr>
+<td align="left"></td>
+<td align="right"><div class="copyright-footer">Copyright © 2001-2008 Joel de Guzman,
+ Hartmut Kaiser
+ Distributed under the Boost Software License, Version 1.0. (See accompanying
+ file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
+ 
+</div></td>
+</tr></table>
+<hr>
+<div class="spirit-nav">
+<a accesskey="p" href="lexer_quickstart2.html"><img src="../../../../../../../doc/html/images/prev.png" alt="Prev"></a><a accesskey="u" href="../tutorials.html"><img src="../../../../../../../doc/html/images/up.png" alt="Up"></a><a accesskey="h" href="../../../index.html"><img src="../../../../../../../doc/html/images/home.png" alt="Home"></a><a accesskey="n" href="../abstracts.html"><img src="../../../../../../../doc/html/images/next.png" alt="Next"></a>
+</div>
+</body>
+</html>

Modified: trunk/libs/spirit/doc/html/spirit/qi_and_karma.html
==============================================================================
--- trunk/libs/spirit/doc/html/spirit/qi_and_karma.html (original)
+++ trunk/libs/spirit/doc/html/spirit/qi_and_karma.html 2008-07-14 21:20:34 EDT (Mon, 14 Jul 2008)
@@ -120,7 +120,7 @@
<div class="titlepage"><div><div><h4 class="title">
<a name="spirit.qi_and_karma.tutorials.quick_start"></a>Quick Start</h4></div></div></div>
<a name="spirit.qi_and_karma.tutorials.quick_start.why_would_you_want_to_use_spirit_qi_"></a><h5>
-<a name="id459151"></a>
+<a name="id642887"></a>
 <a href="qi_and_karma.html#spirit.qi_and_karma.tutorials.quick_start.why_would_you_want_to_use_spirit_qi_">Why
 would you want to use Spirit.Qi?</a>
 </h5>
@@ -167,7 +167,7 @@
 on how to build parsers from the simplest parser, building up as we go.
 
<a name="spirit.qi_and_karma.tutorials.warming_up.trivial_example__1_parsing_a_number"></a><h5>
-<a name="id459523"></a>
+<a name="id642932"></a>
 <a href="qi_and_karma.html#spirit.qi_and_karma.tutorials.warming_up.trivial_example__1_parsing_a_number">Trivial
 Example #1 Parsing a number</a>
 </h5>
@@ -183,7 +183,7 @@
 parsers and consistent naming conventions help you keep from going insane!
 
<a name="spirit.qi_and_karma.tutorials.warming_up.trivial_example__2_parsing_two_numbers"></a><h5>
-<a name="id385053"></a>
+<a name="id642962"></a>
 <a href="qi_and_karma.html#spirit.qi_and_karma.tutorials.warming_up.trivial_example__2_parsing_two_numbers">Trivial
 Example #2 Parsing two numbers</a>
 </h5>
@@ -217,7 +217,7 @@
 </td></tr>
</table></div>
<a name="spirit.qi_and_karma.tutorials.warming_up.trivial_example__3_parsing_one_or_more_numbers"></a><h5>
-<a name="id385164"></a>
+<a name="id643025"></a>
 <a href="qi_and_karma.html#spirit.qi_and_karma.tutorials.warming_up.trivial_example__3_parsing_one_or_more_numbers">Trivial
 Example #3 Parsing one or more numbers</a>
 </h5>
@@ -242,7 +242,7 @@
 in most places.
 
<a name="spirit.qi_and_karma.tutorials.warming_up.trivial_example__4_parsing_a_comma_delimited_list_of_numbers"></a><h5>
-<a name="id459640"></a>
+<a name="id643070"></a>
 <a href="qi_and_karma.html#spirit.qi_and_karma.tutorials.warming_up.trivial_example__4_parsing_a_comma_delimited_list_of_numbers">Trivial
 Example #4 Parsing a comma-delimited list of numbers</a>
 </h5>
@@ -267,7 +267,7 @@
 star encloses the complete expression above.
 
<a name="spirit.qi_and_karma.tutorials.warming_up.let_s_parse_"></a><h5>
-<a name="id459835"></a>
+<a name="id643204"></a>
 <a href="qi_and_karma.html#spirit.qi_and_karma.tutorials.warming_up.let_s_parse_">Let's
 Parse!</a>
 </h5>
@@ -637,7 +637,7 @@
 The examples above can be found here: ../../example/qi/actions.cpp
 
<a name="spirit.qi_and_karma.tutorials.semantic_actions.phoenix"></a><h5>
-<a name="id462372"></a>
+<a name="id644996"></a>
 <a href="qi_and_karma.html#spirit.qi_and_karma.tutorials.semantic_actions.phoenix">Phoenix</a>
 </h5>

@@ -1195,7 +1195,7 @@
 </li>
</ul></div>
<a name="spirit.qi_and_karma.tutorials.roman_numerals.symbol_table"></a><h5>
-<a name="id466741"></a>
+<a name="id649273"></a>
 <a href="qi_and_karma.html#spirit.qi_and_karma.tutorials.roman_numerals.symbol_table">Symbol
 Table</a>
 </h5>
@@ -1330,7 +1330,7 @@
 They are all parsers.
 
<a name="spirit.qi_and_karma.tutorials.roman_numerals.rules"></a><h5>
-<a name="id468029"></a>
+<a name="id650781"></a>
 <a href="qi_and_karma.html#spirit.qi_and_karma.tutorials.roman_numerals.rules">Rules</a>
 </h5>

@@ -1417,7 +1417,7 @@
r = double_ >> *(',' >> double_);
</pre>
<a name="spirit.qi_and_karma.tutorials.roman_numerals.grammars"></a><h5>
-<a name="id468623"></a>
+<a name="id651182"></a>
 <a href="qi_and_karma.html#spirit.qi_and_karma.tutorials.roman_numerals.grammars">Grammars</a>
 </h5>

@@ -1517,7 +1517,7 @@
 </li>
</ul></div>
<a name="spirit.qi_and_karma.tutorials.roman_numerals.let_s_parse_"></a><h5>
-<a name="id469487"></a>
+<a name="id651798"></a>
 <a href="qi_and_karma.html#spirit.qi_and_karma.tutorials.roman_numerals.let_s_parse_">Let's
 Parse!</a>
 </h5>
@@ -1721,7 +1721,7 @@
 has a <tt class="computeroutput">std::string</tt> attribute.
 
<a name="spirit.qi_and_karma.tutorials.employee___parsing_into_structs.lexeme"></a><h5>
-<a name="id471572"></a>
+<a name="id653876"></a>
 <a href="qi_and_karma.html#spirit.qi_and_karma.tutorials.employee___parsing_into_structs.lexeme">Lexeme</a>
 </h5>
<pre class="programlisting">
@@ -1740,7 +1740,7 @@
 a double quote.
 
<a name="spirit.qi_and_karma.tutorials.employee___parsing_into_structs.difference"></a><h5>
-<a name="id471735"></a>
+<a name="id653986"></a>
 <a href="qi_and_karma.html#spirit.qi_and_karma.tutorials.employee___parsing_into_structs.difference">Difference</a>
 </h5>

@@ -1761,7 +1761,7 @@
 is just <tt class="computeroutput">char</tt>.
 
<a name="spirit.qi_and_karma.tutorials.employee___parsing_into_structs.plus"></a><h5>
-<a name="id471899"></a>
+<a name="id654092"></a>
 <a href="qi_and_karma.html#spirit.qi_and_karma.tutorials.employee___parsing_into_structs.plus">Plus</a>
 </h5>
<pre class="programlisting">
@@ -1783,7 +1783,7 @@
std::vector<char>
</pre>
<a name="spirit.qi_and_karma.tutorials.employee___parsing_into_structs.sequence_attribute"></a><h5>
-<a name="id472091"></a>
+<a name="id654219"></a>
 <a href="qi_and_karma.html#spirit.qi_and_karma.tutorials.employee___parsing_into_structs.sequence_attribute">Sequence
 Attribute</a>
 </h5>
@@ -1827,7 +1827,7 @@
 </td></tr>
</table></div>
<a name="spirit.qi_and_karma.tutorials.employee___parsing_into_structs.attribute_collapsing"></a><h5>
-<a name="id472417"></a>
+<a name="id654444"></a>
 <a href="qi_and_karma.html#spirit.qi_and_karma.tutorials.employee___parsing_into_structs.attribute_collapsing">Attribute
 Collapsing</a>
 </h5>
@@ -1862,7 +1862,7 @@
std::vector<char>
</pre>
<a name="spirit.qi_and_karma.tutorials.employee___parsing_into_structs.auto_rules"></a><h5>
-<a name="id472772"></a>
+<a name="id654684"></a>
 <a href="qi_and_karma.html#spirit.qi_and_karma.tutorials.employee___parsing_into_structs.auto_rules">Auto
 Rules</a>
 </h5>
@@ -1904,7 +1904,7 @@
 directly into the rule's attribute, in-situ.
 
<a name="spirit.qi_and_karma.tutorials.employee___parsing_into_structs.finally"></a><h5>
-<a name="id473227"></a>
+<a name="id656165"></a>
 <a href="qi_and_karma.html#spirit.qi_and_karma.tutorials.employee___parsing_into_structs.finally">Finally</a>
 </h5>

@@ -2015,6 +2015,9 @@
<li>
 Not Predicate
 </li>
+<li>
+ Lazy Lit
+ </li>
</ul></div>

 The full cpp files for these examples can be found here: ../../example/qi/mini_xml1.cpp
@@ -2025,7 +2028,7 @@
 for testing purposes. "4.toyxml" has an error in it.
 
<a name="spirit.qi_and_karma.tutorials.mini_xml___asts_.first_cut"></a><h5>
-<a name="id474137"></a>
+<a name="id656817"></a>
 <a href="qi_and_karma.html#spirit.qi_and_karma.tutorials.mini_xml___asts_.first_cut">First
 Cut</a>
 </h5>
@@ -2096,7 +2099,7 @@
 by the placeholder <tt class="computeroutput">_val</tt>).
 
<a name="spirit.qi_and_karma.tutorials.mini_xml___asts_.alternates"></a><h5>
-<a name="id475429"></a>
+<a name="id657775"></a>
 <a href="qi_and_karma.html#spirit.qi_and_karma.tutorials.mini_xml___asts_.alternates">Alternates</a>
 </h5>
<pre class="programlisting">
@@ -2130,7 +2133,7 @@
;
</pre>
<a name="spirit.qi_and_karma.tutorials.mini_xml___asts_.not_predicate"></a><h5>
-<a name="id475897"></a>
+<a name="id658115"></a>
 <a href="qi_and_karma.html#spirit.qi_and_karma.tutorials.mini_xml___asts_.not_predicate">Not
 Predicate</a>
 </h5>
@@ -2159,7 +2162,7 @@
 at this point.
 
<a name="spirit.qi_and_karma.tutorials.mini_xml___asts_.inherited_attribute"></a><h5>
-<a name="id476145"></a>
+<a name="id658956"></a>
 <a href="qi_and_karma.html#spirit.qi_and_karma.tutorials.mini_xml___asts_.inherited_attribute">Inherited
 Attribute</a>
 </h5>
@@ -2187,7 +2190,7 @@
 <tt class="computeroutput">_r3</tt>, etc. if you have more).
 
<a name="spirit.qi_and_karma.tutorials.mini_xml___asts_.a_lazy_lit"></a><h5>
-<a name="id476429"></a>
+<a name="id659154"></a>
 <a href="qi_and_karma.html#spirit.qi_and_karma.tutorials.mini_xml___asts_.a_lazy_lit">A
 Lazy Lit</a>
 </h5>
@@ -2223,7 +2226,7 @@
 accessors, provided by Phoenix.
 
<a name="spirit.qi_and_karma.tutorials.mini_xml___asts_.how_it_all_works"></a><h5>
-<a name="id476912"></a>
+<a name="id659495"></a>
 <a href="qi_and_karma.html#spirit.qi_and_karma.tutorials.mini_xml___asts_.how_it_all_works">How
 it all works</a>
 </h5>
@@ -2272,7 +2275,7 @@
 as <tt class="computeroutput">_r1</tt>.
 
<a name="spirit.qi_and_karma.tutorials.mini_xml___asts_.the_structures"></a><h5>
-<a name="id477360"></a>
+<a name="id659801"></a>
 <a href="qi_and_karma.html#spirit.qi_and_karma.tutorials.mini_xml___asts_.the_structures">The
 Structures</a>
 </h5>
@@ -2306,7 +2309,7 @@

 
<a name="spirit.qi_and_karma.tutorials.mini_xml___asts_.of_alternates_and_variants"></a><h5>
-<a name="id477635"></a>
+<a name="id659999"></a>
 <a href="qi_and_karma.html#spirit.qi_and_karma.tutorials.mini_xml___asts_.of_alternates_and_variants">Of
 Alternates and Variants</a>
 </h5>
@@ -2335,7 +2338,7 @@
 of <tt class="computeroutput">a</tt> and <tt class="computeroutput">B</tt> is the attribute of <tt class="computeroutput">b</tt>.
 
<a name="spirit.qi_and_karma.tutorials.mini_xml___asts_.adapting_structs_again"></a><h5>
-<a name="id477907"></a>
+<a name="id660177"></a>
 <a href="qi_and_karma.html#spirit.qi_and_karma.tutorials.mini_xml___asts_.adapting_structs_again">Adapting
 structs again</a>
 </h5>
@@ -2362,7 +2365,7 @@

 
<a name="spirit.qi_and_karma.tutorials.mini_xml___asts_.one_more_take"></a><h5>
-<a name="id478089"></a>
+<a name="id660305"></a>
 <a href="qi_and_karma.html#spirit.qi_and_karma.tutorials.mini_xml___asts_.one_more_take">One
 More Take</a>
 </h5>
@@ -2426,7 +2429,7 @@
 in the <tt class="computeroutput">xml</tt> rule:
 
<a name="spirit.qi_and_karma.tutorials.mini_xml___asts_.local_variables"></a><h5>
-<a name="id479056"></a>
+<a name="id661021"></a>
 <a href="qi_and_karma.html#spirit.qi_and_karma.tutorials.mini_xml___asts_.local_variables">Local
 Variables</a>
 </h5>
@@ -2555,7 +2558,7 @@
 What's new?
 
<a name="spirit.qi_and_karma.tutorials.mini_xml___error_handling.readable_names"></a><h5>
-<a name="id480845"></a>
+<a name="id663414"></a>
 <a href="qi_and_karma.html#spirit.qi_and_karma.tutorials.mini_xml___error_handling.readable_names">Readable
 Names</a>
 </h5>
@@ -2576,7 +2579,7 @@
end_tag.name("end_tag");
</pre>
<a name="spirit.qi_and_karma.tutorials.mini_xml___error_handling.on_error"></a><h5>
-<a name="id481100"></a>
+<a name="id663594"></a>
 <a href="qi_and_karma.html#spirit.qi_and_karma.tutorials.mini_xml___error_handling.on_error">On
 Error</a>
 </h5>
@@ -2756,7 +2759,7 @@
</tbody>
</table></div>
<a name="spirit.qi_and_karma.tutorials.mini_xml___error_handling.expectation_points"></a><h5>
-<a name="id481774"></a>
+<a name="id664105"></a>
 <a href="qi_and_karma.html#spirit.qi_and_karma.tutorials.mini_xml___error_handling.expectation_points">Expectation
 Points</a>
 </h5>

Modified: trunk/libs/spirit/doc/html/spirit/qi_and_karma/abstracts/mini_xml___error_handling.html
==============================================================================
--- trunk/libs/spirit/doc/html/spirit/qi_and_karma/abstracts/mini_xml___error_handling.html (original)
+++ trunk/libs/spirit/doc/html/spirit/qi_and_karma/abstracts/mini_xml___error_handling.html 2008-07-14 21:20:34 EDT (Mon, 14 Jul 2008)
@@ -111,7 +111,7 @@
 What's new?
 
<a name="spirit.qi_and_karma.abstracts.mini_xml___error_handling.readable_names"></a><h5>
-<a name="id484146"></a>
+<a name="id666234"></a>
 <a href="mini_xml___error_handling.html#spirit.qi_and_karma.abstracts.mini_xml___error_handling.readable_names">Readable
 Names</a>
 </h5>
@@ -132,7 +132,7 @@
end_tag.name("end_tag");
</pre>
<a name="spirit.qi_and_karma.abstracts.mini_xml___error_handling.on_error"></a><h5>
-<a name="id484400"></a>
+<a name="id666414"></a>
 <a href="mini_xml___error_handling.html#spirit.qi_and_karma.abstracts.mini_xml___error_handling.on_error">On
 Error</a>
 </h5>
@@ -312,7 +312,7 @@
</tbody>
</table></div>
<a name="spirit.qi_and_karma.abstracts.mini_xml___error_handling.expectation_points"></a><h5>
-<a name="id485068"></a>
+<a name="id666926"></a>
 <a href="mini_xml___error_handling.html#spirit.qi_and_karma.abstracts.mini_xml___error_handling.expectation_points">Expectation
 Points</a>
 </h5>

Modified: trunk/libs/spirit/doc/html/spirit/qi_and_karma/abstracts/parsing.html
==============================================================================
--- trunk/libs/spirit/doc/html/spirit/qi_and_karma/abstracts/parsing.html (original)
+++ trunk/libs/spirit/doc/html/spirit/qi_and_karma/abstracts/parsing.html 2008-07-14 21:20:34 EDT (Mon, 14 Jul 2008)
@@ -56,27 +56,27 @@
 main reason for Spirit's extensibility.
 
<a name="spirit.qi_and_karma.abstracts.parsing.the_api_functions_exposed_by__emphasis_spirit_qi__emphasis__"></a><h5>
-<a name="id482262"></a>
+<a name="id664965"></a>
 <a href="parsing.html#spirit.qi_and_karma.abstracts.parsing.the_api_functions_exposed_by__emphasis_spirit_qi__emphasis__">The
 API functions exposed by Spirit.Qi </a>
 </h5>
<a name="spirit.qi_and_karma.abstracts.parsing.the_parse___function"></a><h5>
-<a name="id482294"></a>
+<a name="id664985"></a>
 <a href="parsing.html#spirit.qi_and_karma.abstracts.parsing.the_parse___function">The
 parse() function</a>
 </h5>
<a name="spirit.qi_and_karma.abstracts.parsing.the_phrase_parse___function"></a><h5>
-<a name="id482323"></a>
+<a name="id665002"></a>
 <a href="parsing.html#spirit.qi_and_karma.abstracts.parsing.the_phrase_parse___function">The
 phrase_parse() function</a>
 </h5>
<a name="spirit.qi_and_karma.abstracts.parsing.the_tokenize_and_parse___function"></a><h5>
-<a name="id482350"></a>
+<a name="id665015"></a>
 <a href="parsing.html#spirit.qi_and_karma.abstracts.parsing.the_tokenize_and_parse___function">The
 tokenize_and_parse() function</a>
 </h5>
<a name="spirit.qi_and_karma.abstracts.parsing.the_tokenize_and_phrase_parse___function"></a><h5>
-<a name="id482380"></a>
+<a name="id665028"></a>
 <a href="parsing.html#spirit.qi_and_karma.abstracts.parsing.the_tokenize_and_phrase_parse___function">The
 tokenize_and_phrase_parse() function</a>
 </h5>

Modified: trunk/libs/spirit/doc/html/spirit/qi_and_karma/abstracts/parsing_and_generating.html
==============================================================================
--- trunk/libs/spirit/doc/html/spirit/qi_and_karma/abstracts/parsing_and_generating.html (original)
+++ trunk/libs/spirit/doc/html/spirit/qi_and_karma/abstracts/parsing_and_generating.html 2008-07-14 21:20:34 EDT (Mon, 14 Jul 2008)
@@ -29,27 +29,27 @@
 and Generating">Parsing
 and Generating</a></h4></div></div></div>
<a name="spirit.qi_and_karma.abstracts.parsing_and_generating.the_api_functions_exposed_by__emphasis_spirit_qi__emphasis__"></a><h5>
-<a name="id482437"></a>
+<a name="id665052"></a>
 <a href="parsing_and_generating.html#spirit.qi_and_karma.abstracts.parsing_and_generating.the_api_functions_exposed_by__emphasis_spirit_qi__emphasis__">The
 API functions exposed by Spirit.Qi </a>
 </h5>
<a name="spirit.qi_and_karma.abstracts.parsing_and_generating.the_parse___function"></a><h5>
-<a name="id482472"></a>
+<a name="id665069"></a>
 <a href="parsing_and_generating.html#spirit.qi_and_karma.abstracts.parsing_and_generating.the_parse___function">The
 parse() function</a>
 </h5>
<a name="spirit.qi_and_karma.abstracts.parsing_and_generating.the_phrase_parse___function"></a><h5>
-<a name="id482503"></a>
+<a name="id665082"></a>
 <a href="parsing_and_generating.html#spirit.qi_and_karma.abstracts.parsing_and_generating.the_phrase_parse___function">The
 phrase_parse() function</a>
 </h5>
<a name="spirit.qi_and_karma.abstracts.parsing_and_generating.the_tokenize_and_parse___function"></a><h5>
-<a name="id482532"></a>
+<a name="id665096"></a>
 <a href="parsing_and_generating.html#spirit.qi_and_karma.abstracts.parsing_and_generating.the_tokenize_and_parse___function">The
 tokenize_and_parse() function</a>
 </h5>
<a name="spirit.qi_and_karma.abstracts.parsing_and_generating.the_tokenize_and_phrase_parse___function"></a><h5>
-<a name="id482563"></a>
+<a name="id665109"></a>
 <a href="parsing_and_generating.html#spirit.qi_and_karma.abstracts.parsing_and_generating.the_tokenize_and_phrase_parse___function">The
 tokenize_and_phrase_parse() function</a>
 </h5>

Modified: trunk/libs/spirit/doc/html/spirit/qi_and_karma/quick_reference.html
==============================================================================
--- trunk/libs/spirit/doc/html/spirit/qi_and_karma/quick_reference.html (original)
+++ trunk/libs/spirit/doc/html/spirit/qi_and_karma/quick_reference.html 2008-07-14 21:20:34 EDT (Mon, 14 Jul 2008)
@@ -46,7 +46,7 @@
</dl>
</div>
<div class="table">
-<a name="id485566"></a>Table 4. Character Parsers
+<a name="id669006"></a>Table 4. Character Parsers
<table class="table" summary="Character Parsers">
<colgroup>
<col>

Modified: trunk/libs/spirit/doc/html/spirit/qi_and_karma/reference.html
==============================================================================
--- trunk/libs/spirit/doc/html/spirit/qi_and_karma/reference.html (original)
+++ trunk/libs/spirit/doc/html/spirit/qi_and_karma/reference.html 2008-07-14 21:20:34 EDT (Mon, 14 Jul 2008)
@@ -50,7 +50,7 @@
<div class="titlepage"><div><div><h5 class="title">
<a name="spirit.qi_and_karma.reference.concepts.parser"></a>Parser</h5></div></div></div>
<a name="spirit.qi_and_karma.reference.concepts.parser.description"></a><h6>
-<a name="id486482"></a>
+<a name="id669736"></a>
 <a href="reference.html#spirit.qi_and_karma.reference.concepts.parser.description">Description</a>
 </h6>

@@ -66,7 +66,7 @@
</dl>
</div>
<a name="spirit.qi_and_karma.reference.concepts.parser.valid_expressions"></a><h6>
-<a name="id486542"></a>
+<a name="id669786"></a>
 <a href="reference.html#spirit.qi_and_karma.reference.concepts.parser.valid_expressions">Valid
 Expressions</a>
 </h6>
@@ -126,7 +126,7 @@
</tr></tbody>
</table></div>
<a name="spirit.qi_and_karma.reference.concepts.parser.type_requirements"></a><h6>
-<a name="id486680"></a>
+<a name="id669898"></a>
 <a href="reference.html#spirit.qi_and_karma.reference.concepts.parser.type_requirements">Type
 Requirements</a>
 </h6>
@@ -161,14 +161,14 @@
</tr></tbody>
</table></div>
<a name="spirit.qi_and_karma.reference.concepts.parser.invariants"></a><h6>
-<a name="id486784"></a>
+<a name="id669984"></a>
 <a href="reference.html#spirit.qi_and_karma.reference.concepts.parser.invariants">Invariants</a>
 </h6>

 For any Parser xxx the following invariants always hold:
 
<a name="spirit.qi_and_karma.reference.concepts.parser.models"></a><h6>
-<a name="id486815"></a>
+<a name="id670004"></a>
 <a href="reference.html#spirit.qi_and_karma.reference.concepts.parser.models">Models</a>
 </h6>

Modified: trunk/libs/spirit/doc/html/spirit/qi_and_karma/reference/debug.html
==============================================================================
--- trunk/libs/spirit/doc/html/spirit/qi_and_karma/reference/debug.html (original)
+++ trunk/libs/spirit/doc/html/spirit/qi_and_karma/reference/debug.html 2008-07-14 21:20:34 EDT (Mon, 14 Jul 2008)
@@ -7,7 +7,7 @@
<link rel="start" href="../../../index.html" title="Spirit 2.0">
<link rel="up" href="../reference.html" title="Reference">
<link rel="prev" href="auxiliary.html" title="Auxiliary">
-<link rel="next" href="../../__lex__.html" title="Spirit.Lex">
+<link rel="next" href="../../lex.html" title=" Spirit.Lex">
</head>
<body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF">
<table cellpadding="2" width="100%"><tr>
@@ -20,7 +20,7 @@
</tr></table>
<hr>
<div class="spirit-nav">
-<a accesskey="p" href="auxiliary.html"><img src="../../../../../../../doc/html/images/prev.png" alt="Prev"></a><a accesskey="u" href="../reference.html"><img src="../../../../../../../doc/html/images/up.png" alt="Up"></a><a accesskey="h" href="../../../index.html"><img src="../../../../../../../doc/html/images/home.png" alt="Home"></a><a accesskey="n" href="../../__lex__.html"><img src="../../../../../../../doc/html/images/next.png" alt="Next"></a>
+<a accesskey="p" href="auxiliary.html"><img src="../../../../../../../doc/html/images/prev.png" alt="Prev"></a><a accesskey="u" href="../reference.html"><img src="../../../../../../../doc/html/images/up.png" alt="Up"></a><a accesskey="h" href="../../../index.html"><img src="../../../../../../../doc/html/images/home.png" alt="Home"></a><a accesskey="n" href="../../lex.html"><img src="../../../../../../../doc/html/images/next.png" alt="Next"></a>
</div>
<div class="section" lang="en"><div class="titlepage"><div><div><h4 class="title">
<a name="spirit.qi_and_karma.reference.debug"></a>Debug</h4></div></div></div></div>
@@ -35,7 +35,7 @@
</tr></table>
<hr>
<div class="spirit-nav">
-<a accesskey="p" href="auxiliary.html"><img src="../../../../../../../doc/html/images/prev.png" alt="Prev"></a><a accesskey="u" href="../reference.html"><img src="../../../../../../../doc/html/images/up.png" alt="Up"></a><a accesskey="h" href="../../../index.html"><img src="../../../../../../../doc/html/images/home.png" alt="Home"></a><a accesskey="n" href="../../__lex__.html"><img src="../../../../../../../doc/html/images/next.png" alt="Next"></a>
+<a accesskey="p" href="auxiliary.html"><img src="../../../../../../../doc/html/images/prev.png" alt="Prev"></a><a accesskey="u" href="../reference.html"><img src="../../../../../../../doc/html/images/up.png" alt="Up"></a><a accesskey="h" href="../../../index.html"><img src="../../../../../../../doc/html/images/home.png" alt="Home"></a><a accesskey="n" href="../../lex.html"><img src="../../../../../../../doc/html/images/next.png" alt="Next"></a>
</div>
</body>
</html>

Modified: trunk/libs/spirit/doc/html/spirit/what_s_new.html
==============================================================================
--- trunk/libs/spirit/doc/html/spirit/what_s_new.html (original)
+++ trunk/libs/spirit/doc/html/spirit/what_s_new.html 2008-07-14 21:20:34 EDT (Mon, 14 Jul 2008)
@@ -26,7 +26,7 @@
<div class="titlepage"><div><div><h2 class="title" style="clear: both">
<a name="spirit.what_s_new"></a>What's New</h2></div></div></div>
<a name="spirit.what_s_new.spirit_classic"></a><h3>
-<a name="id456029"></a>
+<a name="id638679"></a>
<a href="what_s_new.html#spirit.what_s_new.spirit_classic">Spirit Classic</a>
</h3>

Modified: trunk/libs/spirit/doc/lex.qbk
==============================================================================
--- trunk/libs/spirit/doc/lex.qbk (original)
+++ trunk/libs/spirit/doc/lex.qbk 2008-07-14 21:20:34 EDT (Mon, 14 Jul 2008)
@@ -6,18 +6,18 @@
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]

-[section __lex__]
+[section:lex __lex__]

[include lex/introduction.qbk]

-[section __lex__ Tutorials]
+[section:tutorials __lex__ Tutorials]
[include lex/lexer_tutorials.qbk]
[include lex/lexer_quickstart1.qbk]
[include lex/lexer_quickstart2.qbk]
[include lex/lexer_quickstart3.qbk]
[endsect]

-[section Abstracts]
+[section:abstracts Abstracts]
[section Lexer Primitives]
[include lex/lexer_primitives.qbk]
[include lex/tokens_values.qbk]
@@ -31,11 +31,11 @@
[include lex/lexer_states.qbk]
[endsect]

-[section Quick Reference]
+[section:quick_reference Quick Reference]
[endsect]

-[section Reference]
-[section Concepts]
+[section:reference Reference]
+[section:concepts Concepts]
[include reference/lex/lexer.qbk]
[include reference/lex/token.qbk]
[include reference/lex/tokendef.qbk]

Modified: trunk/libs/spirit/doc/lex/introduction.qbk
==============================================================================
--- trunk/libs/spirit/doc/lex/introduction.qbk (original)
+++ trunk/libs/spirit/doc/lex/introduction.qbk 2008-07-14 21:20:34 EDT (Mon, 14 Jul 2008)
@@ -6,7 +6,7 @@
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]

-[section Introduction to __lex__]
+[section:lexer_introduction Introduction to __lex__]

Lexical scanning is the process of analyzing the stream of input characters and
separating it into strings called tokens, separated by whitespace.
@@ -49,7 +49,7 @@
* Type 3: Regular grammars

The complexity of these grammars increases from regular grammars being the
-simplest to unrestricted grammars being the most complex. Similarily, the
+simplest to unrestricted grammars being the most complex. Similarly, the
complexity of the recognizers for these grammars increases. Although, a few
features of some programming languages (such as C++) are Type 1, fortunately
for the most part programming languages can be described using only the Types 3
@@ -134,4 +134,21 @@
generators (such as __re2c__, __ragel__, etc.). But it is very flexible and
allows to speed up the development of your application.

+[heading The Library Structure of __lex__]
+
+The [link spirit.lexerflow figure] below shows a high level overview of how the
+__lex__ library might be used in an application. __lex__ allows to create
+lexical analyzers based on patterns. These patterns are regular expression
+based rules used to define the different tokens to be recognized in the
+character input sequence. The input sequence is expected to be provided to the
+lexical analyzer as an arbitrary standard forward iterator. The lexical
+analyzer itself exposes a standard forward iterator as well. The difference
+here is that the exposed iterator provides access to the token sequence instead
+of to the character sequence. The tokens in this sequence are constructed on
+the fly by analyzing the underlying character sequence and
+matching this to the patterns as defined by the application.
+
+[fig ./images/lexerflow.png..The Library structure and Common Flow of Information while using __lex__ in an application..spirit.lexerflow]
+
+
[endsect]

Modified: trunk/libs/spirit/doc/lex/lexer_attributes.qbk
==============================================================================
--- trunk/libs/spirit/doc/lex/lexer_attributes.qbk (original)
+++ trunk/libs/spirit/doc/lex/lexer_attributes.qbk 2008-07-14 21:20:34 EDT (Mon, 14 Jul 2008)
@@ -6,7 +6,7 @@
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]

-[section Lexer Attributes]
+[section:lexer_attributes Lexer Attributes]

[endsect]

Modified: trunk/libs/spirit/doc/lex/lexer_primitives.qbk
==============================================================================
--- trunk/libs/spirit/doc/lex/lexer_primitives.qbk (original)
+++ trunk/libs/spirit/doc/lex/lexer_primitives.qbk 2008-07-14 21:20:34 EDT (Mon, 14 Jul 2008)
@@ -6,9 +6,10 @@
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]

-[section Lexer Primitives]
+[section:lexer_primitives Lexer Primitives]

[/ Describe the primitive lexer constructs, such as token_def, token_set? ]
+
[/ Describe the primitive lexer constructs usable in parsers, such as
in_state[], set_state(), token(), etc. ]

Modified: trunk/libs/spirit/doc/lex/lexer_quickstart1.qbk
==============================================================================
--- trunk/libs/spirit/doc/lex/lexer_quickstart1.qbk (original)
+++ trunk/libs/spirit/doc/lex/lexer_quickstart1.qbk 2008-07-14 21:20:34 EDT (Mon, 14 Jul 2008)
@@ -6,7 +6,7 @@
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]

-[section Quickstart 1 - A word counter using __lex__]
+[section:lexer_quickstart1 Quickstart 1 - A word counter using __lex__]

__lex__ is very modular, which follows the general building principle of the
__spirit__ libraries. You never pay for features you don't use. It is nicely

Modified: trunk/libs/spirit/doc/lex/lexer_quickstart2.qbk
==============================================================================
--- trunk/libs/spirit/doc/lex/lexer_quickstart2.qbk (original)
+++ trunk/libs/spirit/doc/lex/lexer_quickstart2.qbk 2008-07-14 21:20:34 EDT (Mon, 14 Jul 2008)
@@ -6,7 +6,7 @@
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]

-[section Quickstart 2 - A better word counter using __lex__]
+[section:lexer_quickstart2 Quickstart 2 - A better word counter using __lex__]

People knowing __flex__ will probably complain about the example from the
section __sec_lex_quickstart_1__ as being overly complex and not being

Modified: trunk/libs/spirit/doc/lex/lexer_quickstart3.qbk
==============================================================================
--- trunk/libs/spirit/doc/lex/lexer_quickstart3.qbk (original)
+++ trunk/libs/spirit/doc/lex/lexer_quickstart3.qbk 2008-07-14 21:20:34 EDT (Mon, 14 Jul 2008)
@@ -6,7 +6,7 @@
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]

-[section Quickstart 3 - Counting Words Using a Parser]
+[section:lexer_quickstart3 Quickstart 3 - Counting Words Using a Parser]

The whole purpose of integrating __lex__ as part of the __spirit__ library was
to add a library allowing to merge lexical analysis with the parsing

Modified: trunk/libs/spirit/doc/lex/lexer_semantic_actions.qbk
==============================================================================
--- trunk/libs/spirit/doc/lex/lexer_semantic_actions.qbk (original)
+++ trunk/libs/spirit/doc/lex/lexer_semantic_actions.qbk 2008-07-14 21:20:34 EDT (Mon, 14 Jul 2008)
@@ -6,5 +6,5 @@
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]

-[section Lexer Semantic Actions]
+[section:lexer_semantic_actions Lexer Semantic Actions]
[endsect]

Modified: trunk/libs/spirit/doc/lex/lexer_states.qbk
==============================================================================
--- trunk/libs/spirit/doc/lex/lexer_states.qbk (original)
+++ trunk/libs/spirit/doc/lex/lexer_states.qbk 2008-07-14 21:20:34 EDT (Mon, 14 Jul 2008)
@@ -6,7 +6,7 @@
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]

-[section Lexer States]
+[section:lexer_states Lexer States]

[heading Controlling the Lexer State from Lexer Semantic Actions]

Modified: trunk/libs/spirit/doc/lex/lexer_static_model.qbk
==============================================================================
--- trunk/libs/spirit/doc/lex/lexer_static_model.qbk (original)
+++ trunk/libs/spirit/doc/lex/lexer_static_model.qbk 2008-07-14 21:20:34 EDT (Mon, 14 Jul 2008)
@@ -6,7 +6,7 @@
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]

-[section The /Static/ Lexer Model]
+[section:lexer_static_model The /Static/ Lexer Model]

The documentation of __lex__ so far mostly was about describing the features of
the /dynamic/ model, where the tables needed for lexical analysis are generated

Modified: trunk/libs/spirit/doc/lex/lexer_tutorials.qbk
==============================================================================
--- trunk/libs/spirit/doc/lex/lexer_tutorials.qbk (original)
+++ trunk/libs/spirit/doc/lex/lexer_tutorials.qbk 2008-07-14 21:20:34 EDT (Mon, 14 Jul 2008)
@@ -6,7 +6,7 @@
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]

-[section __lex__ Tutorials Overview]
+[section:lexer_tutorials __lex__ Tutorials Overview]

The __lex__ library implements several components on top of possibly different
lexer generator libraries. It exposes a pair of iterators, which, when

Modified: trunk/libs/spirit/doc/lex/parsing_using_a_lexer.qbk
==============================================================================
--- trunk/libs/spirit/doc/lex/parsing_using_a_lexer.qbk (original)
+++ trunk/libs/spirit/doc/lex/parsing_using_a_lexer.qbk 2008-07-14 21:20:34 EDT (Mon, 14 Jul 2008)
@@ -6,7 +6,7 @@
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]

-[section Parsing using a Lexer]
+[section:lexer_parsing Parsing using a Lexer]

[/ write about integration of lexer component with __qi__]

Modified: trunk/libs/spirit/doc/lex/token_definition.qbk
==============================================================================
--- trunk/libs/spirit/doc/lex/token_definition.qbk (original)
+++ trunk/libs/spirit/doc/lex/token_definition.qbk 2008-07-14 21:20:34 EDT (Mon, 14 Jul 2008)
@@ -6,6 +6,6 @@
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]

-[section Ways to define Tokens]
+[section:lexer_token_definition Ways to define Tokens]

[endsect]

Modified: trunk/libs/spirit/doc/lex/tokenizing.qbk
==============================================================================
--- trunk/libs/spirit/doc/lex/tokenizing.qbk (original)
+++ trunk/libs/spirit/doc/lex/tokenizing.qbk 2008-07-14 21:20:34 EDT (Mon, 14 Jul 2008)
@@ -6,7 +6,7 @@
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]

-[section Tokenizing Input Data]
+[section:lexer_tokenizing Tokenizing Input Data]

[heading The tokenize() function]

Modified: trunk/libs/spirit/doc/lex/tokens_values.qbk
==============================================================================
--- trunk/libs/spirit/doc/lex/tokens_values.qbk (original)
+++ trunk/libs/spirit/doc/lex/tokens_values.qbk 2008-07-14 21:20:34 EDT (Mon, 14 Jul 2008)
@@ -6,7 +6,7 @@
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]

-[section About Tokens and Token Values]
+[section:lexer_token_values About Tokens and Token Values]

As already discussed, lexical scanning is the process of analyzing the stream
of input characters and separating it into strings called tokens, most of the

Modified: trunk/libs/spirit/doc/spirit2.qbk
==============================================================================
--- trunk/libs/spirit/doc/spirit2.qbk (original)
+++ trunk/libs/spirit/doc/spirit2.qbk 2008-07-14 21:20:34 EDT (Mon, 14 Jul 2008)
@@ -70,18 +70,18 @@
[def __sec_qi_and_karma__ [link spirit.qi_and_karma Qi and Karma]]
[def __sec_qi_karma_attributes__ [link spirit.qi_and_karma.abstracts.attributes Attributes]]

-[def __sec_lex__ [link spirit.__lex__ Lex]]
-[def __sec_lex_quickstart_1__ [link spirit.__lex__.__lex___tutorials.quickstart_1___a_word_counter_using___lex__ Lex Quickstart 1 - A word counter using __lex__]]
-[def __sec_lex_quickstart_2__ [link spirit.__lex__.__lex___tutorials.quickstart_2___a_better_word_counter_using___lex__ Lex Quickstart 2 - A better word counter using __lex__]]
-[def __sec_lex_quickstart_3__ [link spirit.__lex__.__lex___tutorials.quickstart_3___counting_words_using_a_parser Lex Quickstart 3 - Counting Words Using a Parser]]
-
-[def __sec_lex_static_model__ [link spirit.__lex__.abstracts.the__static__lexer_model The /Static/ Model]]
-[def __sec_lex_primitives__ [link spirit.__lex__.abstracts.lexer_primitives Lexer Primitives]]
-[def __sec_lex_tokenvalues__ [link spirit.__lex__.abstracts.lexer_primitives.about_tokens_and_token_values About Tokens and Token Values]]
-[def __sec_lex_attributes__ [link spirit.__lex__.abstracts.lexer_attributes Lexer Attributes]]
+[def __sec_lex__ [link spirit.lex Lex]]
+[def __sec_lex_quickstart_1__ [link spirit.lex.tutorials.lexer_quickstart1 Lex Quickstart 1 - A word counter using __lex__]]
+[def __sec_lex_quickstart_2__ [link spirit.lex.tutorials.lexer_quickstart2 Lex Quickstart 2 - A better word counter using __lex__]]
+[def __sec_lex_quickstart_3__ [link spirit.lex.tutorials.lexer_quickstart3 Lex Quickstart 3 - Counting Words Using a Parser]]
+
+[def __sec_lex_static_model__ [link spirit.lex.abstracts.lexer_static_model The /Static/ Model]]
+[def __sec_lex_primitives__ [link spirit.lex.abstracts.lexer_primitives Lexer Primitives]]
+[def __sec_lex_tokenvalues__ [link spirit.lex.abstracts.lexer_primitives.lexer_token_values About Tokens and Token Values]]
+[def __sec_lex_attributes__ [link spirit.lex.abstracts.lexer_attributes Lexer Attributes]]

-[def __sec_ref_lex_token__ [link spirit.__lex__.reference.concepts.token Token Reference]]
-[def __sec_ref_lex_token_def__ [link spirit.__lex__.reference.concepts.tokendef TokenDef Reference]]
+[def __sec_ref_lex_token__ [link spirit.lex.reference.concepts.token Token Reference]]
+[def __sec_ref_lex_token_def__ [link spirit.lex.reference.concepts.tokendef TokenDef Reference]]

[/ References to API descriptions ]

@@ -91,11 +91,11 @@

[/ References to classes ]

-[def __class_token_def__ [link spirit.__lex__.reference.tokendef_class `token_def<>`]]
+[def __class_token_def__ [link spirit.lex.reference.tokendef_class `token_def<>`]]

-[def __class_lexertl_token__ [link spirit.__lex__.reference.token_class `lexertl_token<>`]]
-[def __class_lexertl_lexer__ [link spirit.__lex__.reference.lexer_class.the_lexertl_lexer_class_implementing_the_dynamic_model `lexertl_lexer<>`]]
-[def __class_lexertl_static_lexer__ [link spirit.__lex__.reference.lexer_class.the_lexertl_static_lexer_class_implementing_the_static_model `lexertl_static_lexer<>`]]
+[def __class_lexertl_token__ [link spirit.lex.reference.token_class `lexertl_token<>`]]
+[def __class_lexertl_lexer__ [link spirit.lex.reference.lexer_class.the_lexertl_lexer_class_implementing_the_dynamic_model `lexertl_lexer<>`]]
+[def __class_lexertl_static_lexer__ [link spirit.lex.reference.lexer_class.the_lexertl_static_lexer_class_implementing_the_static_model `lexertl_static_lexer<>`]]

[/ Some images ]

Next message: widman_at_[hidden]: "[Boost-commit] svn:boost r47429 - sandbox/committee/concepts/issues"
Previous message: dgregor_at_[hidden]: "[Boost-commit] svn:boost r47427 - branches/CMake/release/libs/accumulators"

Date view	Thread view	Subject view	Author view

Boost-Commit list run by bdawes at acm.org, david.abrahams at rcn.com, gregod at cs.rpi.edu, cpdaniel at pacbell.net, john at johnmaddock.co.uk