Spirit X3 comment parser

From: Olaf Peter (ope-devel_at_[hidden])
Date: 2021-05-17 11:35:24

Hallo Users,

this time a question about retaining the comments in the recursive AST.
I've never seen those question before.

I have a grammar that can have a comment at any point while maintaining
that same grammar. I am interested in the comments to be able to parse
the possibly contained pragma etc. statement later (so parsed as raw).

...
auto const comment_def =
 Â Â Â Â Â Â  x3::lit("//")
 Â Â Â  >> raw[*(char_ - x3::eol) >> x3::eol];
...

The AST is ready, an AST node for the comment is also there. How do I
change the rule now?

auto const start_def =
 Â Â Â  x3::skip(x3::space)[
 Â Â Â Â Â Â Â Â Â  comment
 Â Â Â Â Â Â Â  | toplevel_rule
 Â Â Â  ];

with the actual parse function:

...
x3::parse(iter, end, toplevel_parser >> x3::eoi, toplevel);
 Â ....

originally the latter looked like this:

....
x3::phrase_parse(iter, end, toplevel_parser >> x3::eoi, parser::skipper,
toplevel);
...

The above start_def unfortunately also changes the attribute for it,
ast::toplevel_node now becomes a
variant<ast::comment_node,ast::toplevel_node>. I'm also not sure how
this rule affects the dependent rules, i.e. recursive and low level in
terms of correctness and performance. For the AST visitor this just
means another overload.

Another point is the test suite. Currently I test the low level sub
rules with a separate instance, where the test input can also contain
comments for test case documentation purpose. With x3::phrase_parse()
this was not a problem. With the new approach, for each rule to be
tested, I also have to use a variant <comment,attribute> for the
attribute, which is very inconvenient....

So the question, is this really the right way? Are there alternatives?

Thanks in advance