Re: [Boost-users] [Spirit] Qi lexeme only taking the first word

Subject: Re: [Boost-users] [Spirit] Qi lexeme only taking the first word
From: rmawatson rmawatson (rmawatson_at_[hidden])
Date: 2018-11-07 08:25:36

> this has an attribute of vector<string,vector<tuple<char,std::string>>> or something similar.

Where are you getting that from? It makes no sense whatsoever given
the struct full_it_t { std::string val; }, which is similarly mapped,
and ruled,
I've just had a look and the synthesized attribute is actually
boost::fusion::vector<std::basic_string<char,std::char_traits<char>,std::allocator<char> >,std::vector<boost::fusion::vector<char,std::basic_string<char,std::char_traits<char>,std::allocator<char> > >,std::allocator<boost::fusion::vector<char,std::basic_string<char,std::char_traits<char>,std::allocator<char> > > > > >

Cleaned up that is,

boost::fusion::vector<std::string,std::vector<boost::fusion::vector<char,std::string>>>

So almost exactly what I said. You can get this using spirit::traits::attribute_of.

It makes perfect sense. This is the synthesized attribute of the RHS parser, not the attribute you have passed it. Your struct full_it_t { std::string val; } appears to spirit as boost::fusion::vector<std::string>
this is precisely what the BOOST_FUSION_ADAPT_STRUCT is for. The RHS parsers is the result of the various parsers you use. In your case...

Sequence Parser (a >> b)

Expression --> Attribute
a: A, b: B --> (a >> b): tuple<A, B>

Kleene Parser (*a)
Expression --> Attribute
a: A --> *a: vector<A>

Character Parser (char_, lit)
Expression --> Attribute
ns::char_ --> The character type of the Character Encoding Namespace, ns.

With your expression, where id's attribute is string, you have

id >> *(char_('.') >> id)

gives

tuple<string,vector<tuple<char,string>>>

This is then assigned to the target attribute of the LHS rule. which in your setup is, boost::fusion::vector<std::string>

________________________________
From: Boost-users <boost-users-bounces_at_[hidden]> on behalf of Michael Powell via Boost-users <boost-users_at_[hidden]>
Sent: 07 November 2018 02:08
To: boost-users_at_[hidden]
Cc: Michael Powell
Subject: Re: [Boost-users] [Spirit] Qi lexeme only taking the first word

On Tue, Nov 6, 2018 at 8:12 PM rmawatson rmawatson
<rmawatson_at_[hidden]> wrote:
>
> It's been a long while since I've used spirit::qi. But What it looks like is happeneing in your setup is something liek this,
>
> When you have:
>
> qi::rule<It, AST::full_id_t()> full_id;
>
> the attribute is vector<string>
>
> When it matches
>
> id >> *(char_('.') >> id)
>
> this has an attribute of vector<string,vector<tuple<char,std::string>>> or something similar.

Where are you getting that from? It makes no sense whatsoever given
the struct full_it_t { std::string val; }, which is similarly mapped,
and ruled, etc.

> spirit appears to compare your target attribute with the synthesised attribute of the parser and for any (trailing?) members
> of the synthesised attribute that do not match in your attribute, it marks them as unused_type and they are not assigned.

Would I need to do some grouping or something to persuade Spirit to
treat the struct as I've defined and adapted it?

> You can see overload of assign to is used in your example if you breakpoint it -> boost\spirit\home\qi\detail\assign_to.hpp line 399.
>
> It appears in boost\spirit\home\qi\operator\sequence_base.hpp line 74, where the predicate
> traits::attribute_not_unused<Context, Iterator> is passed to spirit::any_if (boost\spirit\home\support\algorithm\any_if.hpp line 186.)
> it will basically discard attributes where the LHS sequence is not matched with the RHS.
>
> You can see this in your example by adding an additional member to
>
>     struct full_id_t {
>         std::string val;
>         std::vector<std::string> others;
>     };
>
>     BOOST_FUSION_ADAPT_STRUCT(AST::full_id_t, val, others)
>
> Your missing bits will appear in this std::vector, as they are now not silently discarded.
> http://coliru.stacked-crooked.com/a/51f16c6deff45309
>
> I think what the problem fundamentally is the attribute propagation is different when you have a string to when you have a vector<string> as in your two examples.
> the first kicks in whatever logic exists to flatten the LHS attribute into a string, the second takes the first element, assigns it
> and marks the rest as unused.
>
> One thing you can do is use qi::as<std::string>()[ id >> *(char_('.') >> id) ] to force conversion of synthesised attribute to a string to happen
> before it is assigned to your attribute.
> http://coliru.stacked-crooked.com/a/6a060343a390f037
>
> I've only had a quick look and this is pretty half hearted analysis. You'll really have to dig deep to find out exactly what is going on, but I suspect
> this is somewhat along the right lines.
> ________________________________
> From: Boost-users <boost-users-bounces_at_[hidden]> on behalf of Michael Powell via Boost-users <boost-users_at_[hidden]>
> Sent: 06 November 2018 23:03
> To: boost-users_at_[hidden]
> Cc: Michael Powell
> Subject: Re: [Boost-users] [Spirit] Qi lexeme only taking the first word
>
> On Tue, Nov 6, 2018 at 5:40 PM Michael Powell <mwpowellhtx_at_[hidden]> wrote:
> >
> > On Tue, Nov 6, 2018 at 5:01 PM Michael Powell <mwpowellhtx_at_[hidden]> wrote:
> > >
> > > Hello,
> > >
> > > I've got a couple of rules that are perplexing to me. First,
> > >
> > > rule<It, std::string(), St> id %= lexeme[qi::alpha >> *char_("A-Za-z0-9_")];
> > >
> > > In and of itself, id is working fine. Then I've got a "full id":
> > >
> > > rule<It, full_id_t(), St> full_id %= id >> *(char_('.') >> id);
> > >
> > > Where:
> > >
> > > struct full_id_t {
> > >     std::string val;
> > > };
> > >
> > > full_id_t::val is quite intentional for reasons elsewhere in the grammar.
> > >
> > > The perplexity comes in, it seems lexeme is only shaving off the first
> > > word as the val.
> > >
> > > For instance, parsing "two.oranges.red.test", I receive back "two" in the AST.
> > >
> > > Perhaps I should defer specifying the lexeme part of id until later?
> >
> > I elaborated a little on the "simple" full id sub-grammar, but I
> > cannot repro using the GCC compiler. I'm wondering if this has
> > anything to do with the VS2017 fpos issue?
> >
> > http://coliru.stacked-crooked.com/a/adeb42ce2f19b0fd
> >
> > Or there may be insufficient context in the web compiler to adequately demo.
>
> I got a repro:
>
> http://coliru.stacked-crooked.com/a/069a44296240be7e
>
> Although the reasons as to why I do not know.
>
> It is a difference in attribute synthesis. When full_id synthesizes a
> std::string(), the conversion to full_id_t() "just works" magically.
> I'm guessing by happy accident based on the std::string val being the
> only member (adaptation, etc).
>
> But when I change the synthesis to be its "true" type, that is,
> AST::full_id_t(), suddenly I see the same behavior.
>
> Really and truly, I do not know why. Everything else being equal why
> would one approach be any different than the other?
>
> Anyone with some Spirit, Fusion, AST, insights?
>
> Thanks!
>
> For now, I'll run with it as has been exposed here, but it's a bit
> troubling to me not knowing the difference.
>
> > > Thoughts? Suggestions?
> > >
> > > Thank you!
> > >
> > > Best regards,
> > >
> > > Michael Powell
> _______________________________________________
> Boost-users mailing list
> Boost-users_at_[hidden]
> https://lists.boost.org/mailman/listinfo.cgi/boost-users
_______________________________________________
Boost-users mailing list
Boost-users_at_[hidden]
https://lists.boost.org/mailman/listinfo.cgi/boost-users