- Boost - lists.boost.org

Late review of Boost.Parser
by David Sankel 02 Mar '24

02 Mar '24

# Are you knowledgeable about the problem domain? Yes. I've written several parser combinator libraries in C++ and used still more in C++ (Spirit v1, v2, and qi) and other programming langauges (Haskell's Parsec, attoparsec, and others). # How much effort did you put into your evaluation? A glance? A quick reading? In-depth study? I read through the tutorial documentation and some of the reference documentation. I also tried to create several examples with the library and made comparisons to Haskell's Parsec. # What is your evaluation of the potential usefulness of the library? Very # What is your evaluation of the documentation? The documentation isn't great. Consider char_ <https://tzlaine.github.io/parser/doc/html/boost/parser/char_.html>'s documentation. It includes no examples and fails to mention it can be used without arguments. It seems strange that the tutorial focuses so much on semantic actions. It seems like the more common case would be to parse into some kind of abstract syntax tree. # What is your evaluation of the design? Much of the wealth of research into parser combinators wasn't incorporated into this library. I didn't see in the documentation, for example, an easy way to "map" one parser's attribute into another. This kind of thing should be a basis operation provided by the library. # What is your evaluation of the implementation? The usage of has_include is a ODR nightmare for large codebases. It would be better to generate some kind of config.hpp that sets these definitions at compile time. For custom variant/tuple types, user specialization of variable templates is also an ODR nightmare. Again, a config.hpp would be better for this. For this, though, I don't think the configurability is worth the complexity. We have standard types for these so we should use them. # Did you try to use the library? With what compiler? Did you have any problems? Yes, I tried to use the library with a modern compiler on a MacOS machine. I didn't have any problems building. # Do you think the library should be accepted as a Boost library? No, I do not think it should be accepted. Overall, I think the syntax has too much focus on magically filling in user-provided structures instead of the basics of monadic parser combinators and basis operations. Consider the following Haskell parser which evaluates simple parenthesized sum expressions. I wasn't able to use Boost.Parser to accomplish this after reading the documentation and several attempts. I'm sure it's possible, but I don't see how it can be done using the combiners and primitives provided. import Text.Parsec.String (Parser)import Text.Parsec integer :: Parser Integerinteger = do n <- many1 digit return (read n) integerPlus :: Parser IntegerintegerPlus = do x <- integer y <- (try $ char '+' >> expression) <|> return 0 return $ x+y parentheses :: Parser Integerparentheses = do char '(' x <- expression char ')' return x expression :: Parser Integerexpression = integerPlus <|> parentheses

5 9

Boost Tools
by Robert Ramey 01 Mar '24

01 Mar '24

For some time I've felt that our Boost tooling could be improved. By boost tooling I mean things like b2, boost unit testing and integration testing, documentation generation, reports like dependencies, website, etc. Currently we rely on certain privileged authors to develop these things then then upload them to the boost directories. This has worked, but it could be better. We spend what seems to me an inordinate amount of time chasing down anomalies, researching errors, responding for requests for help, etc. I think design, robustness, documentation could be improved. I think that the the most natural way for us to do that would be to adopt the practices that we have used successfully for decades to develop superior quality libraries to the design and development of tooling. That is: a) Someone proposes a new tool or improvement in an existing one on this list. His proposal includes a working though likely incomplete prototype, documentation, tests, demo etc. b) This is subjected to the same formal review process that libraries are. c) Subject to this review, the proposal is either rejected, accepted or accepted with conditions. Of course this would only apply to new tools and updates of existing ones. Robert Ramey

1 0

The Review for Zach's parser library has concluded
by Marshall Clow 01 Mar '24

01 Mar '24

Thanks to everyone who participated, and especially to those to submitted reviews. (And Zach, of courser) I’ll have results soon. — Marshall

1 0

Boost.Parser parsing length-prefixed strings efficiently
by Mohammad Nejati [ashtum] 01 Mar '24

01 Mar '24

Hi everyone, I have two questions about the following code, which attempts to parse length-prefixed strings using the proposed Boost.Parser: ``` std::string input = "[5]HELLO[5]WORLD"; unsigned int size = 0; auto action = [&size](auto & ctx) { size = bp::_attr(ctx); }; auto lps_parser = '[' >> bp::uint_[action] >> ']' >> bp::repeat(std::ref(size))[bp::char_]; auto result = bp::parse(input, *lps_parser); ``` 1. Is it possible to eliminate the need for the action and size variable by using some kind of placeholder in bp::repeat that utilizes the attribute of the first parser? 2. How can we make this part of the parser more efficient: bp::repeat(std::ref(size))[bp::char_] The current implementation seems to loop and execute bp::char_ parser on each character, while all it needs to do is chunk a portion of the input string. Regards, Mohammad Nejati

2 1

New website is open-source!
by Vinnie Falco 29 Feb '24

29 Feb '24

The new website and documentation are provided under the BSL, and the source repositories have been transferred to the Boost GitHub organization: https://github.com/boostorg/website-v2 https://github.com/boostorg/website-v2-docs While we sort out the unresolved issues related to the licensing of artwork and logo, the new website has moved to this temporary location for now: https://boost.io Please continue to experience, enjoy, and most importantly test the new website. A gentle reminder: it is not “finished” and likely will not be “perfect” for a while. The launch of the new website is the beginning of the journey rather than the end. Issues opened here will be acted upon by our dedicated website development team: https://github.com/boostorg/website-v2/issues For documentation-related issues, please open and monitor issues here: https://github.com/boostorg/website-v2-docs/issues Pull requests are always welcomed! Thanks

6 6

Boost.Parser questions
by Ruben Perez 29 Feb '24

29 Feb '24

Hi all, Sorry for jumping into this so late. I'll send my review tomorrow (that's still on time, isn't it?). I've been playing with the library and some questions popped up (please note I've no prior experience with Spirit): * What's the rationale for using operator[] for semantic actions (instead of a regular method like .action())? It looks like the parsing problem with inline lambdas could be easily worked around with this. * What's the rationale for using operator[] for directives? repeat(24)[p] looks like an over-complication, where a regular function repeat(24, p) would have just worked fine. * For non-recursive rules, what's the rationale on preferring the macro over something like: constexpr bp::rule<struct replacement_field_tag, std::string> replacement_field { "replacement_field", +bp::char_ }; * My use case involves parsing identifiers that can only contain ASCII lowercase, uppercase, digits and the underscore. I'd usually do this by defining a `bool is_identifier_char(char c)` function. This is what I could do with Boost.Parser - is there a better way (one that doesn't force me to list all valid characters)? constexpr const char name_chars[] = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ_0123456789"; constexpr auto name_continuation_rng = std::span<const char>( name_continuation, sizeof(name_continuation) - 1 ); * What's the rationale behind the special naming convention for functions like _attr, _value, _globals and the like? Is there a reason why these can't be member functions of the context? * I've found really frustrating the generic lambda approach for semantic actions, since this completely inhibits tooling like clangd. Is there a reason why we can't have a public definition of the parsing context I could use in my functions? * I started my evaluation using clang-18 (Linux) but builds fail (seems to happen under all clang versions I tried). gcc-12 seems to work fine. Note that I'm using the develop branch, since I understand it contains the bug fixes for the issues that previous reviewers found. Regards, Ruben.

6 9

Boost.Parser review
by Richard 29 Feb '24

29 Feb '24

- Do you think the library should be accepted as a Boost library? Yes, this would be a great addition to Boost as a successor to Spirit. ====================================================================== Details: Note: I've only had time to read through the documentation. I'm familar with Spirit Classic, Spirit V2 and X3 parsing libraries and have done presentations on them using my own written example code. - What is your evaluation of the design? It's very similar to Boost.Spirit in philosophy, but other than the (optional) connection to Boost.Hana feels very decoupled from all the other bits of boost at a user level. Spirit's direct connection to things like Phoenix and Fusion made Spirit more complicated to explain to other developers and more difficult to use IMO. I understand why such things were done at the time of Boost.Spirit's writing, but being a C++17 library frees Boost.Parser from all these digressions which is good. The only thing obviously lacking were the binary parsers from Spirit V2 (bug filed). - What is your evaluation of the implementation? I've only reviewed the documentation, so I haven't had any time to dig into the implementation before the review deadline. I will say that there are no comments in the header files and that could be improved. It means that without consulting the tutorial documentation, I have no information in the included headers to guide me. - What is your evaluation of the documentation? The documentation is well written and follows a logical progression. It could use a math expression example as this is one of the most common reasons for needing to write a parser (typical C style math operators and precedence rules). The JSON example is nice (shows recursive rule parsing), but doesn't deal with operator precedence at all, so another detailed example showing this would be helpful. There were some minor typos and mistakes; I filed github issues on those. The concepts and reference section are just raw code dumps. They could use some additional doxygen style comments to tie things together. - What is your evaluation of the potential usefulness of the library? I would find this library a great successor to Spirit. - Did you try to use the library? With what compiler? Did you have any problems? No, I only had time to read through documentation. - How much effort did you put into your evaluation? A glance? A quick reading? In-depth study? I carefully read through every page of the documentation in the Tutorial section and skimmed over the reference section. - Are you knowledgeable about the problem domain? Yes. -- "The Direct3D Graphics Pipeline" free book <http://tinyurl.com/d3d-pipeline> The Terminals Wiki <http://terminals-wiki.org> The Computer Graphics Museum <http://ComputerGraphicsMuseum.org> Legalize Adulthood! (my blog) <http://LegalizeAdulthood.wordpress.com>

3 2

Boost.Parser review
by Ruben Perez 29 Feb '24

29 Feb '24

Hi all, This is my review of the proposed Boost.Parser. As in other occasions, I've taken a practical approach and used the library to (re)implement an existing, real feature in Boost.MySQL: client-side SQL formatting. This requires parsing a format string with replacement fields (similar to std::format strings). - What is your evaluation of the potential usefulness of the library? I think it's useful. Parsing is a common task, and this library simplifies it. I tend to prefer hand-crafted parsers in general, but libraries like the proposed tend to save time (and bugs). Also, many other users have manifested interest in it. I think the library is too heavyweight regarding compile times to be used in header-only libraries like Boost.MySQL, but it should be fine in compiled libraries and applications. - What is your evaluation of the design? I think it's pretty good in general. It allows you to express complex parsers using little code, which is good. In my opinion, it may be too generic (allowing any forward character range as input, for instance), which tends to increase complexity. I tend to prefer simpler approaches like the one Boost.URL uses - but I understand not everyone thinks like me, and there's a use case for each approach. As we discussed in previous posts, if I was writing this, I would have made some decisions differently (see https://lists.boost.org/Archives/boost/2024/02/256010.php) but I don't think there is any problem with the current design. The author has justified his decisions, and everything makes sense. - What is your evaluation of the implementation? I can tell it's really complex. I've attempted to debug a problem I had for which parser::trace::on wasn't enough (see below), and navigating the code was extremely difficult (reminds me of debugging Asio). This is probably inherent to the level of complexity of the library. This is not a criticism, but a thought I wanted to share. - Did you try to use the library? With what compiler? Did you have any problems? Yes, I did. As mentioned above, I re-implemented Boost.MySQL SQL formatting (see https://www.boost.org/doc/libs/master/libs/mysql/doc/html/mysql/sql_formatt…) using the proposed library. I've uploaded the code to a branch - you can check it here: https://github.com/boostorg/mysql/blob/66c99e31c9e548b6cf5b98e3d08255f22390… For reference, I've also pasted the grammar and the code at the end of this email. I've found the following problems: * Rule parsers using sequence + optional parsers yield unexpected results when the optional rule doesn't match. Making it a regular parser, rather than a rule parser, fixes the problem. The impact is that automatic replacement fields (i.e. `{}`) were being parsed as `{0}`. https://github.com/tzlaine/parser/issues/125 tracks this. * My clang-11 build fails. Looks like a concept is being used without proper protection, and the compiler doesn't support it. https://github.com/tzlaine/parser/issues/129 * My clang-16 build (and my local build attempts with clang 17, 18 and 19) fail because of a non-matching template redeclaration in parser/detail/text/transcode_view.hpp. I don't know if it's clang being picky or a clang bug. https://github.com/tzlaine/parser/issues/130 * My clang-14 build fails. This looks like it's hitting a clang + libstdc++ bug. See https://github.com/tzlaine/parser/issues/131. * My Apple clang 14 build fails. It also looks like an issue in clang or libc++. See https://github.com/tzlaine/parser/issues/133. * My MSVC 14.2 and 14.3 builds under C++17 fail (C++20 works fine). I've completely failed to parse what's the problem here. Description here: https://github.com/tzlaine/parser/issues/132 Some other minor problems: * As reported by other reviewers, enabling warnings=all or warnings=extra issues a lot of warnings. * [[no_unique_address]] (C++20) is used unprotected, which causes warnings under clang-7 (see https://github.com/tzlaine/parser/issues/128) * The CMakeLists.txt is not Boost-compliant (yet). * There is a #include <boost/hana.hpp>, which is not very friendly for compile-times - better replace it by the individual header that is needed. I've submitted the build logs together with all details I've been able to gather. I haven't been able to provide a minimal reproducible example for some of these (they were a lot and I had limited time). Note that all the builds referenced in the issues (except the OSX one) are run in Docker containers. If you have access to Docker, reproducing these environments should be trivial. Since I've done this in the past, should the author need any help with this, please let me know, I'm happy to help. Reading the CI files for the proposed library, I think compiler coverage is insufficient - for instance, there are no Linux clang builds, and the only Apple clang build uses clang 13 and C++17. - What is your evaluation of the documentation? The presentation is really good. I struggled to understand the concepts at first because I had no previous experience with parsers. I ended up reading the Bison concepts section (https://www.gnu.org/software/bison/manual/bison.html#Concepts) after which everything made perfect sense. I'd advise to expand the introduction a bit, or just point to other material for people like me. I fail to understand what RESOLVE means in https://tzlaine.github.io/parser/doc/html/boost_parser__proposed_/tutorial/…. Docs state "RESOLVE() is a notional macro that expands to the resolution of parse argument or evaluation of a parse predicate", but I'd say this resolution hasn't been explained yet - a link here would help. I've found the reference to be more confusing that helpful - it looks like a straight header dump. This is made worse by the concepts not being linked. Although it has many terrible things, the quickbook toolchain is (partly) capable of linking concepts - have a look at https://www.boost.org/doc/libs/master/libs/mysql/doc/html/mysql/ref/boost__…. This still has limitations, but I'd say it's better than what's currently there. If this is something you'd be interested in, please let me know and I will help you with it. - How much effort did you put into your evaluation? A glance? A quick reading? In-depth study? I've spent around 2 full days of work reading about the problem domain, studying the docs, writing my use case, diagnosing the problems I reported and writing this review. - Are you knowledgeable about the problem domain? I've never used a parsing library like this before. I've read about the problem in the past, but wasn't very knowledgeable about it before the review. I've written small parsers like the one for format strings in the past, though. - My conclusion My recommendation is to REJECT the library in its current form. This is not a hard rejection - I completely think this library can be useful to people. But I found too many problems while using it - too many to get it into Boost in its current form. I'd be happy to see this library being proposed again in a couple of months, once all the issues found during the review get fixed. - Affiliation disclosure Please note that I'm affiliated with the C++ Alliance. I'd like to thank Zach for proposing the library, and Marshall for managing the review. Regards, Ruben. - Reference: parsing code in Boost.MySQL /** * format-str = *( text | literal-opening-brace | literal-closing-brace | replacement-field ) * text = *( any characters except {} ) * replacement-field = "{" [ replacement-id ] "}" * replacement-id = ushort | field-name * field-name = (character a-z, A-Z, _) *( any characters 0-9, a-z, A-Z, _ ) */ // Note: globals is a format_state structure, which has member functions // to append raw text and replacement fields to the output string // Actions constexpr auto action_text = [](auto& ctx) { _globals(ctx).append_text(_attr(ctx)); }; constexpr auto action_replacement_field = [](auto& ctx) { const std::optional<std::variant<unsigned short, std::string>>& val = _attr(ctx); format_state& st = _globals(ctx); if (val) { if (const auto* idx = std::get_if<unsigned short>(&*val)) { st.append_indexed_field(*idx); } else { st.append_named_field(std::get<std::string>(*val)); } } else { st.append_auto_field(); } }; constexpr auto action_opening_brace = [](auto& ctx) { _globals(ctx).append_text("{"); }; constexpr auto action_closing_brace = [](auto& ctx) { _globals(ctx).append_text("}"); }; // Character ranges constexpr std::array<const char, 63> name_chars{ 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z', 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z', '_', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9' }; constexpr boost::span<const char, 53> name_start_chars{name_chars.data(), 53}; // Parsers and rules constexpr auto field_name = parser::char_(name_start_chars) >> *(parser::char_(name_chars)); constexpr auto replacement_id = parser::ushort_ | field_name; constexpr auto replacement_field = ('{' >> -replacement_id) > '}'; constexpr auto literal_open_brace = parser::lit("{{"); constexpr auto literal_close_brace = parser::lit("}}"); constexpr auto text = +(parser::char_ - '{' - '}'); constexpr auto format_str_parser = *( text[action_text] | literal_open_brace[action_opening_brace] | literal_close_brace[action_closing_brace] | replacement_field[action_replacement_field] ); // Top-level parsing function inline void parse_format_str(string_view format_str, format_context_base& ctx, span<const format_arg> args) { // By default, parser prints to cerr, which we don't want parser::callback_error_handler handler([](const std::string&) {}); // Create a parsing state format_state st(ctx, args); // Invoke the actual parsing bool result = parser::parse( format_str, parser::with_globals(parser::with_error_handler(format_str_parser, handler), st) ); // If there was an error, report it if (!result) { ctx.add_error(client_errc::format_string_invalid_syntax); } }

2 1

[release] The 1.85.0 release will close for major changes TOMORROW
by Marshall Clow 28 Feb '24

28 Feb '24

Calendar here: https://www.boost.org/development/ Next deadline: The master branch will close for the beta next Wednesday. — The release managers

1 0

Boost.Parser review by Tobias
by tobi＠die-loews.de 27 Feb '24

27 Feb '24

Boost.Parser review Why do we need a DSEL-Parser library in Boost: I think nobody, who once has written a parser with a DSEL (like e.g. Spirit) in C++, ever want's to go back to out off source parser-generators like yacc/bison or ANTLR (or even hand written parsers): it is so convenient having the parser definition as a grammar within the code. Additionally, you can quite easily hook into the parser (e.g. Spirit's Phoenix functions). So, IMHO there is definitely need for such C++ libraries. On the other hand, the task of creating such a library is very challenging and requires high expertise and coding standards, as library-users have to rely on its correctness. So, Boost with its standards and peer-review process is definitely the place for such a library. (Personally, I wouldn't use a parser-library the hasn't gone through such a process.) Should accept Boost.Parser into Boost? There are already three parser libraries in Boost: Spirt and its successors Qi and X3 - Spirit: very outdated interface, IIRC C++98 compatible, superseded by Qi - Qi: Simply a revolution in parsing! It was blowing my mind, when I first used it. The way attributes were composed and semantic actions could applied with Phoenix was totally exciting. But of course, there was also the downside of (very) long compilation-times and frustrating long error-messages when something got wrong. And (again IIRC) Qi came to life before C++11, so it is also C++98. - X3: I only wrote one larger X3 parser, and to be frank, IMHO it never made it to production-level. There are several things that worked perfectly with Qi, which I never got running the same way in X3. Additionally, the Spirit libraries seem no longer to be in active development. Now, let's get to Boost.Parser: First, I don't want to go into the details of the library, as I think other reviewers did a very good job there. I've been following its evolving since I heard of it in 2022. Back then, I did some test with it and reported some Issues, which got resolved. During the last two years many additional features got added to it. So, Boost.Parser is definitely under active development. Furthermore, the author is an expert for Unicode, which is reflected in the built-in Unicode support of the library. It is written in modern C++ (requires C++17). It is compile-time friendly, supports std::ranges and concepts. It offers an option/abstraction to choose between std::tuples, if needed, or the more flexible hana-implementation. It's syntax is close to Qi, so transition for Qi-users should not be too hard. The documentation contains a lot of examples that help to understand how the library works and how it can be used. Finally, I really like the very easy to use trace-mode. Again, should accept Boost.Parser into Boost? Definitely Yes! Tobias

3 2