Re: [Boost-bugs] [Boost C++ Libraries] #5566: Spirit.Lex converting a token to its value in a lexer semantic action fails

Subject: Re: [Boost-bugs] [Boost C++ Libraries] #5566: Spirit.Lex converting a token to its value in a lexer semantic action fails
From: Boost C++ Libraries (noreply_at_[hidden])
Date: 2011-05-27 01:21:23

#5566: Spirit.Lex converting a token to its value in a lexer semantic action fails
------------------------------------------+---------------------------------
  Reporter:  Reko Tiira <reko@…>          |       Owner:  hkaiser
      Type:  Bugs                         |      Status:  new    
 Milestone:  To Be Determined             |   Component:  spirit 
   Version:  Boost 1.46.1                 |    Severity:  Problem
Resolution:                               |    Keywords:         
------------------------------------------+---------------------------------

Comment (by hkaiser):

 Replying to [ticket:5566 Reko Tiira <reko@…>]:

 It does not exactly work as you expect as the type of teh token value is a
 variant (initially holding a pair of iterators pointing to the matched
 input sequence). You need to explicitly convert this iterator range into
 the attribute type. Normally this is done while parsing on first access to
 the attribute. Here is your modified code doing this:

 {{{
 #include <string>
 #include <vector>
 #include <boost/spirit/include/lex_lexertl.hpp>
 #include <boost/spirit/include/phoenix_core.hpp>
 #include <boost/spirit/include/phoenix_stl.hpp>

 namespace lex = boost::spirit::lex;
 namespace phoenix = boost::phoenix;


 struct get_string_impl
 {
     template <typename Value>
     struct result
     {
         typedef std::string type;
     };

     template <typename Value>
     std::string operator()(Value const& val) const
     {
         // let Spirit transform the token value (here a variant) into a
 string
         // at this point the variant holds a pair of iterators
         typedef boost::iterator_range<char const*> iterpair_type;
         iterpair_type const& ip = boost::get<iterpair_type>(val);

         return std::string(ip.begin(), ip.end());
     }
 };

 boost::phoenix::function<get_string_impl> get_string;

 template <typename Lexer>
 struct text_decorator_lexer : lex::lexer<Lexer>
 {
         text_decorator_lexer()
                 : tag_open("\\{\\{.+?\\}\\}")
                 , tag_close("\\{\\{\\/.+?\\}\\}")
                 , var("\\$\\$.+?\\$\\$")
                 , any(".")
         {
                 using boost::spirit::lex::_val;
                 using boost::phoenix::push_back;
                 using boost::phoenix::ref;

                 this->self
                         = tag_open  [push_back(ref(tokens),
 get_string(_val))]
                         | tag_close [push_back(ref(tokens),
 get_string(_val))]
                         | var       [push_back(ref(tokens),
 get_string(_val))]
                         | any       [push_back(ref(tokens),
 get_string(_val))]
                         ;
         }

         std::vector<std::string> tokens;
         lex::token_def<std::string> tag_open, tag_close, var, any;
 };

 int main()
 {
         typedef lex::lexertl::token<
             const char*,
             boost::mpl::vector<std::string>,
             boost::mpl::false_
         > token_type;
         typedef lex::lexertl::actor_lexer<token_type> lexer_type;

         text_decorator_lexer<lexer_type> text_decorator_tokenizer;
         std::string input = "Hello {{b}}there{{/b}}, the time is
 $$time$$.";

         char const* first = input.c_str();
         char const* last = &first[input.size()];
         lex::tokenize(first, last, text_decorator_tokenizer);
 }
 }}}

 But to be honest, I needed to fix a (unrelated) problem to make this code
 work, therefore you'll need to update from SVN or wait for Boost V1.47.

-- 
Ticket URL: <https://svn.boost.org/trac/boost/ticket/5566#comment:2>
Boost C++ Libraries <http://www.boost.org/>
Boost provides free peer-reviewed portable C++ source libraries.