Subject: Re: [Boost-bugs] [Boost C++ Libraries] #5566: Spirit.Lex converting a token to its value in a lexer semantic action fails
From: Boost C++ Libraries (noreply_at_[hidden])
Date: 2011-05-27 01:21:23
#5566: Spirit.Lex converting a token to its value in a lexer semantic action fails
------------------------------------------+---------------------------------
Reporter: Reko Tiira <reko@â¦> | Owner: hkaiser
Type: Bugs | Status: new
Milestone: To Be Determined | Component: spirit
Version: Boost 1.46.1 | Severity: Problem
Resolution: | Keywords:
------------------------------------------+---------------------------------
Comment (by hkaiser):
Replying to [ticket:5566 Reko Tiira <reko@â¦>]:
It does not exactly work as you expect as the type of teh token value is a
variant (initially holding a pair of iterators pointing to the matched
input sequence). You need to explicitly convert this iterator range into
the attribute type. Normally this is done while parsing on first access to
the attribute. Here is your modified code doing this:
{{{
#include <string>
#include <vector>
#include <boost/spirit/include/lex_lexertl.hpp>
#include <boost/spirit/include/phoenix_core.hpp>
#include <boost/spirit/include/phoenix_stl.hpp>
namespace lex = boost::spirit::lex;
namespace phoenix = boost::phoenix;
struct get_string_impl
{
template <typename Value>
struct result
{
typedef std::string type;
};
template <typename Value>
std::string operator()(Value const& val) const
{
// let Spirit transform the token value (here a variant) into a
string
// at this point the variant holds a pair of iterators
typedef boost::iterator_range<char const*> iterpair_type;
iterpair_type const& ip = boost::get<iterpair_type>(val);
return std::string(ip.begin(), ip.end());
}
};
boost::phoenix::function<get_string_impl> get_string;
template <typename Lexer>
struct text_decorator_lexer : lex::lexer<Lexer>
{
text_decorator_lexer()
: tag_open("\\{\\{.+?\\}\\}")
, tag_close("\\{\\{\\/.+?\\}\\}")
, var("\\$\\$.+?\\$\\$")
, any(".")
{
using boost::spirit::lex::_val;
using boost::phoenix::push_back;
using boost::phoenix::ref;
this->self
= tag_open [push_back(ref(tokens),
get_string(_val))]
| tag_close [push_back(ref(tokens),
get_string(_val))]
| var [push_back(ref(tokens),
get_string(_val))]
| any [push_back(ref(tokens),
get_string(_val))]
;
}
std::vector<std::string> tokens;
lex::token_def<std::string> tag_open, tag_close, var, any;
};
int main()
{
typedef lex::lexertl::token<
const char*,
boost::mpl::vector<std::string>,
boost::mpl::false_
> token_type;
typedef lex::lexertl::actor_lexer<token_type> lexer_type;
text_decorator_lexer<lexer_type> text_decorator_tokenizer;
std::string input = "Hello {{b}}there{{/b}}, the time is
$$time$$.";
char const* first = input.c_str();
char const* last = &first[input.size()];
lex::tokenize(first, last, text_decorator_tokenizer);
}
}}}
But to be honest, I needed to fix a (unrelated) problem to make this code
work, therefore you'll need to update from SVN or wait for Boost V1.47.
-- Ticket URL: <https://svn.boost.org/trac/boost/ticket/5566#comment:2> Boost C++ Libraries <http://www.boost.org/> Boost provides free peer-reviewed portable C++ source libraries.
This archive was generated by hypermail 2.1.7 : 2017-02-16 18:50:06 UTC