- Boost - lists.boost.org

[async-mqtt5] Formal review starts today
by Klemens Morgenstern 28 Oct '24

28 Oct '24

Hi all, the formal boost review of the async-mqtt5 library strats today, the 16th of October and will last until and including the 25th of this month. Async.MQTT5 is a professional, industrial-grade C++17 client built on Boost.Asio. This Client is designed for publishing or receiving messages from an MQTT 5.0 compatible Broker. Async.MQTT5 represents a comprehensive implementation of the MQTT 5.0 protocol standard, offering full support for publishing or receiving messages with QoS 0, 1, and 2. The library has been developed by mireo, a GPS/GIS software development company, and their CTO Ivica Siladić submitted it. The code under review is the current master, which is tagged as v1.0.2: https://github.com/mireo/async-mqtt5/releases/tag/v1.0.2 The documentation can be found here: https://spacetime.mireo.com/async-mqtt5/ Please explicitly state that you either *accept* or *reject* the inclusion of this library into boost. Also please indicate the time & effort spent on the evaluation and give the reasons for your decision. - Does this library bring real benefit to C++ developers for real world use-case? - Do you have an application for this library? - Does the API match with current best practices? - Is the documentation helpful and clear? - Did you try to use it? What problems or surprises did you encounter? - What is your evaluation of the implementation? - Do you have experience with asio? - Did you work with MQTT in the past? More information about the Boost Formal Review Process can be found at: http://www.boost.org/community/reviews.html The review is open to anyone who is prepared to put in the work of evaluating and reviewing the library. Prior experience in contributing to Boost reviews is not a requirement. Reviews can also be submitted privately to me, and I will not disclose who sent them. I might however cite passages from those reviews in my conclusion. I think I should mention that there's another mqtt library that has been proposed for boost, but has yet to find a review manager: https://github.com/redboltz/async_mqtt Some ideas of how we could review them together have been discussed on the mailing list, but this is an independent review of mireo's async-mqtt5. You don't have to consider redboltz/async_mqtt when writing your review. Thanks to Mireo & Ivica for submitting this library.

8 17

[async-mqtt5] Review ends tomorrow
by Klemens Morgenstern 27 Oct '24

27 Oct '24

Reminder: the review period for async-mqtt5 will end tomorrow on Monday the 28th. Thanks for all the great reviews received so far!

1 0

async-mqtt5 review
by Christian Mazakas 25 Oct '24

25 Oct '24

I'm going to vote that we REJECT async-mqtt5 from Boost. My primary reasoning is that so far almost all the review discussion I've seen is focusing solely on the Asio aspect which makes this a miss. I like Asio, and I've used it to deliver products as well. But I don't think this is the MQTT library for Boost. I think libraries like these are a mistake because they're so heavily focused on the ceremony of Asio that the actual meat and potatoes of the library is lost, which in this case is supposed to be MQTT parsing and serialization and broker management. I think there's a lot of utility in MQTT, but what I'd like to see in the future for libraries like these is a first-class interface _without_ I/O. For example, I'm not sure I saw any fuzzing of the underlying parsing code. A non-I/O based interface makes this trivial. I also have a lot of concerns here because the library uses Asio in its interfaces and it also internally uses Spirit. I haven't tried to compile anything but I can only imagine the binary size and the compilation times for a realistic project using this code. I don't have a lot of experience with MQTT as a protocol but my operating assumption is that if TLS/SSL can be made without I/O, so can anything else. I think exploring this design space will lead to a much better outcome for a library going into Boost. You can see prior art here via rustls, OpenSSL's BIO pairs, and Botan which uses a synchronous callback-based interface for handling things. By focusing on creating a pure protocol library, you give users the ability to add MQTT to their own I/O layers which has the extensibility and genericity I'd expect from a Boost library. It will also likely enable the library to be compiled which in this case would be a huge boon. For example, I have an asynchronous runtime which is dramatically faster than Asio and I'm capable of handling low-level networking coding but I can't use this library with my own. Instead, I'd have to roll my own MQTT support. In the ideal case, I'd be able to treat this library like I would Botan and I'd add MQTT support non-intrusively. All this being said, a protocol library that comes with an Asio-based impl is perfectly fine. My only qualm is when the library _only_ supports Asio as seems to be the case here. - Christian

8 14

[async-mqtt5] Review questions
by Ruben Perez 25 Oct '24

25 Oct '24

Hi Ivica, First of all, thanks for submitting your library to Boost. I've been reading the discussion and reference and have some questions for you: 1. The limitation listed on https://spacetime.mireo.com/async-mqtt5/async_mqtt5/auto_reconnect.html about concealing configuration issues looks concerning, as it can be very frustrating for the user. As far as I know, Boost.Redis solved it by including logging as part of its API, and Boost.MySQL did it by providing an extra diagnostics output argument, populated on operation cancellation, that informs the user of potential causes to the problem. Neither of us are very happy with our solution, to be frank. But I think it's a point that might be worth solving before going into Boost. As an option, I have been looking into abusing Asio handler tracking (https://live.boost.org/doc/libs/1_86_0/doc/html/boost_asio/overview/core/ha…) to log custom events. BOOST_ASIO_HANDLER_OPERATION might be a possibility. What are your thoughts regarding this? 2. Quoting the allocators section (https://spacetime.mireo.com/async-mqtt5/async_mqtt5/asio_compliance/allocat…) "[...] associates the allocator of the first packet in the queue to the low-level Boost.Asio function async_write on the transport layer". Does this mean that the queue might use the allocator bound to the first async_publish operation that is issued? If this is the case, wouldn't it be violating the requirement that all memory allocated by an associated allocator is deallocated before the async operation completes? 3. Quoting the executors section (https://spacetime.mireo.com/async-mqtt5/async_mqtt5/asio_compliance/executo…) "The same executor must execute mqtt_client::async_run and all the subsequent async_xxx operations". This sounds like an unusual restriction. What happens if it is violated? Wouldn't it make more sense to use the executor associated with async_run for intermediate handlers, and then the async_xxx associated executor for the final one? 4. I noticed that async_mqtt5 does not implement persistent and clean sessions. I'm no MQTT expert, so I can't evaluate whether these features are relevant or not - I just know that they exist. Is there a reason why you chose to not implement them? If so, I'd suggest adding these reasons somewhere in the docs. 5. According to what I read in the docs, message payloads might contain UTF-8 encoded text or just data blobs, depending on the payload_format_indicator property passed to publish. However, std::string is always used for payloads. As I understand it, if I want to send a binary blob, I need to copy my blob to a string (probably with a reinterpret_cast in the way), and then setting payload_format_indicator. Would it make sense to expose a simpler API taking a blob (e.g. a std::vector<unsigned char>) to make this use case safer? 6. I can't find any docs (other than the examples) on how to use TLS. >From the simplest example (https://spacetime.mireo.com/async-mqtt5/async_mqtt5/hello_world_over_tls.ht…) it looks like setting up TLS requires defining some non-trivial template specializations. I'd like these to be documented. On the other hand, the specializations seem to be using only Boost.Asio types, rather than user-defined types. What would happen if two libraries separately define these hooks because they need MQTT over TLS? Wouldn't it make more sense to provide an optional header providing these functions? 7. What's the use of the TlsContext type in mqtt_client? It looks very coupled to asio::ssl::stream. 8. I've noticed that async_receive returns with an error code whenever a reconnection occurs, which forces the user to re-subscribe to topics. This contrasts with how async_publish works, where reconnection is transparent to the user. Is there a reason for this difference? Would it make sense to implement automatic resubscription on reconnection? Regards, Ruben.

8 28

Summer Mentorship / Volunteering
by Vinnie Falco 24 Oct '24

24 Oct '24

Do any library authors or maintainers: * Desire to mentor a student, and * have the skill to mentor a student, and * have appropriate student work in one of their libraries? If yes please reply with the library and the nature of the work :) Thanks

2 1

[async-mqtt5] Mohammad's review
by Mohammad Nejati [ashtum] 23 Oct '24

23 Oct '24

Hi everyone, This is my review of the proposed async_mqtt5 library. First, I want to thank the Mireo team for submitting this library and Klemens for managing the review process. > Does this library bring real benefit to C++ developers for real world use-case? Developing a high-quality MQTT client library is no small task, and it is not something everyone wants to take on alongside their main project. Having a well-implemented and tested library would be highly valuable for C++ developers. > Do you have an application for this library? > Did you work with MQTT in the past? I have limited experience with MQTT from a previous job where I used the mqtt_cpp library to transmit and receive telemetry data in the telecommunications industry. > Does the API match with current best practices? The API simplifies library usage by providing high-level features, such as automatic reconnection, which is often essential in platforms where MQTT is used. The initiator functions adhere to Asio best practices, allowing users to provide their own executor and allocator, and include proper cancellation support. I believe there’s room to add interfaces for bulk receive and publish operations, which could enhance performance by minimizing rescheduling. This could be considered in the future based on user demand. `mqtt_client` is templated on the stream type and TLS context, which isn't inherently a problem, but it has made Boost.Beast a required dependency, which feels like a design flaw. I suggest abstracting the stream type and its required functionalities into a separate type, allowing users to just include the required implementations of it. I believe `mqtt_client` can type-erase the stream type without any measurable performance impact. All operations on the stream occur at a low frequency, so one level of indirection shouldn't matter. The only higher-frequency operations are `async_write` and `async_read_some`, but since `async_write` uses scatter/gather pattern and passes all buffers in a single call, and `async_read_some` reads into a large buffer, the number of calls to these functions will still be low. I didn’t find the `mqtt_client::brokers` interface very elegant; it could accept a range of hosts and ports and delegate the parsing of a comma-separated list to the users. > Is the documentation helpful and clear? While the documentation does a great job explaining design decisions and corner cases, I'd love to see a beginner-friendly section (like the example in the introduction) that walks through a few examples step by step. This is likely the most-read part of any library documentation and deserves more attention. > Did you try to use it? What problems or surprises did you encounter? I've tried experimenting with the provided examples on a locally run mqtt broker. The obvious problem was that there is no error reporting when things go wrong. This is a side effect of having infinite internal reconnections and no mechanism for reporting them. A logging solution or callbacks for reporting problems would be necessary. > What is your evaluation of the implementation? The implementation has a high quality and shows the authors have a very good understanding of Asio and asynchronous programing. Other than the mentioned issue with stream type, I couldn't find any significant problem during the review. I suggest specializing asio::associator instead implementation associator functions one by one: https://www.boost.org/doc/libs/develop/doc/html/boost_asio/reference/associ… Instead of using the associated allocator of the completion handler to allocate the internal queue, it might be possible to use a form of circular buffer to eliminate the need for frequent memory allocation (this might also fix the total cancellation problem in async_publish). > Do you have experience with asio? Yes, I've been a user of Asio and currently maintain Boost.Beast. > How much time did you spend on this review? I spent about 18 hours reading the documentation, reviewing some parts of the implementation, and experimenting with the interface. > Final decision and comments I vote that we conditionally accept `async_mqtt5` into Boost with the following criteria: - Break the hard dependency on Beast and TLS stream by either abstracting the stream type and placing different implementations in separate headers, or by type-erasing the stream type and constructing the connection through a factory. The type-erasure method is preferred, as it can reduce compile times and simplify usage. - Provide a mechanism for reporting connection errors and reconnection attempts to the user. - Set limits on all internal queues and allow TCP's back-pressure to regulate the speed of the producer and consumer sides. - Introduce address, UB and memory sanitizer to CI builds. Thanks, Mohammad.

3 4

[async-mqtt5] Ruben's review
by Ruben Perez 23 Oct '24

23 Oct '24

Hi all, This is my review of async_mqtt5. First of all, thanks Ivica and the Mireo team for submitting this library for Boost review, and Klemens for managing the review. - Does this library bring real benefit to C++ developers for real world use-case? Yes. MQTT clients are widely used in the IoT world. Boost.Asio is one of the strongest networking libraries in C++. Bringing these together in a high-quality library is valuable. The library already has already more than 140 stars on GitHub, which is a good sign. - Do you have an application for this library? Not right now. But I used to work in the IoT world, and I would have used it if I would have had it available back then. - Does the API match with current best practices? It does almost everywhere. Analysis follows. 1. It's only a high-level API, akin to Boost.Redis. This may not cover some very specialized use cases, but makes it much easier handling common tasks robustly. I don't think this is concerning, since the API is flexible enough. Moreover, the Mireo team has more field experience regarding the protocol than I do. A lower level client can be added in the future without breaking compatibility, if requested by users. 2. It correctly follows Boost.Asio universal async model (except for a gotcha with executors, see my next point), accounting for allocator customization, custom completion tokens, and per-operation cancellation. Looking at the code, it's clear that the developers bear a deep understanding of Asio, which is very valuable. 3. At the moment, the client requires the following: "The same executor must execute mqtt_client::async_run and all the subsequent async_xxx operations." (quoting https://spacetime.mireo.com/async-mqtt5/async_mqtt5/asio_compliance/executo…) This is a deviation from the standard Asio model, not present in any of the libraries currently in Boost. The former statement makes the following malformed, without any diagnostic being emitted: cli.async_run(asio::bind_executor(ex1, ...)); // 1 cli.async_publish(asio::bind_executor(ex2, ...)); // 2 cli.async_publish(asio::bind_executor(ex3, ...)); // 3 Where operations 2 and 3 may complete on any of ex1, ex2 or ex3. I think this limitation should be lifted, and that the above code should be legal. I'd suggest making stream writes child agents of async_run, instead of async_publish and similar functions (see https://lists.boost.org/Archives/boost/2024/10/258132.php for the discussion regarding this issue). 4. The client is templated on the stream type. This is a well-tested design pattern, but can lead to higher compile times. Writing dynamic clients (where the transport type or TLS usage is determined by a runtime variable) is also more involved. I think the current architecture is acceptable, but I'd consider adding a type-erased client in the future. While this shouldn't be too much effort for well-versed Asio developers, regular users may find it unsurmountable. Alternatively, I'd also be good with completely replacing the templated client with a type-erased one (up to the authors). 5. The library internally buffers outgoing and ingoing messages automatically, which is great because it's easier to maintain resilience. However, the only limit on the publishing side are the limited 2^16 packet IDs (restricted by the protocol), with no limit on the reading side. I'm slightly concerned about applications buffering too much on either side. I don't know if this is a real concern or just produced by my lack of MQTT field experience. Would it make sense to somehow introduce a per-topic limit on the number of unacknowledged messages on both the subscribing side (possibly specified when the client subscribes) and on the publishing side? If that's not possible, I'd like to see an example on how to implement this flow control limit on the publishing side, at least. 6. Some APIs use owning types (std::vector and std::string) where non-owning types (boost::core::span and std::string_view) could suffice. I'd suggest reviewing these. 7. TLS streams require a number of template specializations to set up. At the moment, this must be done by the user even for the usual asio::ssl::stream. This risks ODR violations if two libraries end up defining such specializations independently. I'd advise placing the required boilerplate code, together with helper typedefs, in an optional, separate header. 8. The API is really small, and I think that's great. - Is the documentation helpful and clear? It is in general well written and clear. It is however geared towards a user that knows Asio and MQTT pretty well. I'd suggest a couple of changes: 1. There are a lot of pages justifying design decisions made by the library. This has been very useful to me, but is geared towards advanced users. I'd attempt to gather all pages explaining how to use the library in the first section, and move pages with design rationales to a second section. 2. There is no page in the discussion addressing how to use async_publish and async_receive (only an example). This new page should likely go into the first section. 3. There is no page in the discussion addressing TLS. 4. I miss an example about custom authentication. 5. I don't think the multi-threaded example captures the problems that a multi-threaded program integrating the library may face. Running everything under a single strand negates the benefit of multi-threading. I'd target the publisher example, instead. It's important that you ensure that reading the sensor happens outside the strand. I have more minor comments, included at the bottom of this email. - Did you try to use it? What problems or surprises did you encounter? I built a sample publisher and subscriber, based on the examples. They're available here: https://github.com/anarthal/async-mqtt5-test The publisher reads two sensors at different frequencies, which is something that scales poorly when using threads. The async architecture makes this task trivial, which is great. Note that the publisher doesn't co_await async_publish, because that'd prevent further sensor reads, reducing the benefits of resilience. The publisher example in the docs may consider mentioning this. Debugging config issues is extremely difficult (see https://spacetime.mireo.com/async-mqtt5/async_mqtt5/auto_reconnect.html for a rationale). I initially tried to use a public test broker that rejected my connection because of high utilization. It took me a decent amount of time to figure it out - I tried debugging, enabling Asio handler tracking, until I finally launched Wireshark and figured it out. This is a problem during development but also in production, as logging when things go wrong is a usual practice. I think this should be somehow addressed, at the very least by suggesting to use Wireshark/tshark (or other tools that may be practical) in the docs. When using TLS, using SSL_CTX_set_keylog_callback (https://docs.openssl.org/1.1.1/man3/SSL_CTX_set_keylog_callback/#synopsis) to create a keylog file for Wireshark may also be useful, and mentioning how to do it may be useful. I'd at least consider the possibility of including some sort of logging or diagnostics generation, either by using a custom macro or abusing Asio handler tracking. I found that the implementation does not correctly shutdown TLS and websockets (no calls to asio::ssl::stream::async_shutdown and beast::websocket::stream::async_close). These should be added (https://github.com/mireo/async-mqtt5/issues/18) - What is your evaluation of the implementation? The implementation is well written and very easy to follow, which is great. Build times are pretty high, which is not great. The sample publisher takes 22s to build under clang-18, Linux, Debug, C++23 on my machine (30s in Release mode). Building with -ftime-trace, two points show up in the flame graph: 1. The client header includes boost/beast/websocket/stream.hpp, even when not using websockets. The header is very heavyweight and slows down compilation. I think the class can be forward declared if the definition of rebind_executor for websocket streams is changed to the following: template <typename Stream, bool deflate_supported, typename Executor> struct rebind_executor<boost::beast::websocket::stream<asio::ssl::stream<Stream>, deflate_supported>, Executor> { using other = typename boost::beast::websocket::stream< asio::ssl::stream<typename rebind_executor<Stream, Executor>::other>, deflate_supported >; }; Some other Beast headers are included in detail/connect_op.hpp. Connection attempts to distinguish whether the stream is a websocket using metaprogramming, and then runs Beast-specific code when a websocket is detected. I think this creates too much coupling between the libraries. My suggestion is to re-architect this a bit. I'd rewrite websocket detection as a separate trait, and then provide a definition for it in a separate header, thus making Beast a peer dependency. Beast is also use for get_lowest_layer. I don't know if this is strictly required, since asio::ip::tcp::socket does implement the layered protocol (it defines a lowest_layer_type and a get_lowest_layer function, see asio::basic_socket). 2. asio::append and asio::prepend are heavily used, which are heavy to instantiate. I'd suggest experimenting with replacing these by a custom handler class that specializes asio::associator to propagate associated characteristics, as suggested in this ML post: https://lists.boost.org/Archives/boost/2024/10/258132.php. This may or may not reduce compile times, but it's worth a try. The library has many Boost dependencies. While this might not itself impact compile times, it affects package managers. Using Boostdep, the dependency with most transitive dependencies (by far) is Spirit, pulling in variant, thread, preprocessor, phoenix, and a number of other libs. The obvious solution would be encouraging the author to remove the dependency. But I think we have something off in how we manage dependencies in Boost, since the parts of Spirit they're pulling don't require most of these libraries. Regarding testing, I like seeing both unit and integration tests. Code coverage is great. I haven't looked into them in great detail, but from some snippets I saw and Ivica's comments, they seem correctly architectured. I miss sanitizers, fuzz testing and some compilers. See my comments below. - Do you have experience with asio? Yes. I developed and maintain Boost.MySQL. - Did you work with MQTT in the past? I know the basic concepts and have read part of the spec for this review. I haven't used MQTT in production. I have worked in embedded systems before, though. - How much time did you spend on this review? I spent around 25h reading the documentation, reference, researching MQTT, building code snippets and discussing design decisions on the mailing list. - Final decision and comments My recommendation is to CONDITIONALLY ACCEPT async_mqtt5 into Boost. It's a great library and I'd love to see it in Boost. Here are a lot of feedback points, in decreasing order of importance. 1. Acceptance conditions: I think these should be addressed before Boost integration. - Executor correctness. The restriction of forbidding different executors for different operations should be lifted. - Making debugging configuration easier. This can have the form of a documentation page explaining the available tools and how to use them, or by enhancing the current API with logging facilities. - TLS/websocket shutdown code should be added, as per https://github.com/mireo/async-mqtt5/issues/18. 2. Recommendations. These are not acceptance criteria, but I strongly recommend considering them, at least: - Remove the hard dependency on Boost.Beast websocket in detail/rebind_executor.hpp by forward-declaring websocket::stream. - Change the websocket connect handling from if constexpr to traits, to remove the Beast hard dependency. - Consider creating an optional header containing the required boilerplate to use TLS-based connections without the need of template specializations. Helper typedefs could be beneficial. - Experiment with build times and -ftime-trace by reducing the usage of asio::prepend and Boost.Spirit in the implementation. If this happens to improve build times, consider applying the relevant changes. - Introduce fuzz testing for the encoders and decoders (I can aid with this if required). - Introduce address and undefined sanitizer builds (I can also aid with this). - Consider supporting total cancellation in async_publish and other functions. Probably requires fixing the executor issue. - Consider introducing a type-erased client that does not depend on the stream type. I'd consider acceptable entirely removing the templated one in favor of the type-erased one, if the authors feel comfortable with that. - Consider changing the mqtt_client::brokers signature to take a span<host_and_port>, where host_and_port contains a string and a port number. I think this better represents what the function does. - Consider changing usages of std::vector and std::string in public APIs to boost::core::span and std::string_view where it makes sense. - Consider introducing APIs to limit the message production/consumption rates per topic. If not feasible, introduce an example on how to rate-limit a publisher user-side. - Consider trimming some Boost dependencies to make the library lighter for package managers. 3. Minor comments. These are not acceptance criteria, either. - Consider adding CIs for clang 16 to 19. - Apply concept checking to completion tokens, using the BOOST_ASIO_COMPLETION_TOKEN_FOR macro for C++17 compatibility. - Consider using an inline variable to implement get_error_code_category, instead of a static one. - Place non-documented, non-public functions in the detail namespace. Candidate functions: - client_error_to_string, in error.hpp - client_ec_category, in error.hpp - Auxiliary functions and types implementing properties, in property_types.hpp - to_reason_code, in reason_codes.hpp - Consider making the library compatible with the BOOST_ASIO_USE_TS_EXECUTOR_AS_DEFAULT macro. Although legacy, other libraries (namely a postgresql one) require it. - Consider making the will class a plain struct, instead of a class. This would make it compatible with C++20 designated initializers. - Consider renaming the library and namespace to mqtt5 or mqtt, following what we have for MySQL and Redis. Note that typing boost::async_mqtt5 may be long. - Consider making constexpr global variables (like the ones in reason_codes.hpp) inline constexpr. - Consider enhancing support for Asio handler tracking (as per https://live.boost.org/doc/libs/1_86_0/doc/html/boost_asio/overview/core/ha…) by using the BOOST_ASIO_HANDLER_LOCATION macro. This may mitigate the frustration of users experiencing connectivity problems. - Consider running the thread-safety example under the thread sanitizer. I've found a lot of surprises (and learnt a lot) by doing this. - Consider renaming client_service to something that doesn't include the term "service". When working with Asio, a service usually refers to something inheriting from asio::execution_context::service, allocated once per execution context. This is not the case for client_service, which initially created confusing expectations to me. - Consider replacing the SFINAE clause in authenticator() by a static_assert. Calling authenticator() with something not fulfilling the concept is always an error - the clearer the diagnostic, the better. - Consider whether asio::cancel_after/asio::cancel_at can be used to simplify some of the internal waiting logic. - Consider moving the code snippets shown in the discussion to a C++ file and use Quickbook import statements to include them in the docs. This reduces the possibility of having out-of-sync snippets. - Consider specializing asio::associated_executor, asio::associated_immediate_executor and asio::associated_allocator to implement property propagation in detail::cancellable_handler. 4. Comments on docs. These are not acceptance criteria. - The reference lists is_authenticator (https://spacetime.mireo.com/async-mqtt5/async_mqtt5/ref/is_authenticator.ht…) but it looks like it's in the detail namespace. Either promote it to the public namespace, or hide it from the docs (you can still create a page for the concept even if you don't expose the actual type). - The reference lists the phrase "On immediate completion, invocation of the handler will be performed in a manner equivalent to using boost::asio::post.". This is not true, since you support immediate executors. - Some property types are listed as scalars, when they are really vectors of scalars. This might be confusing. For instance, https://spacetime.mireo.com/async-mqtt5/async_mqtt5/ref/publish_props.html lists subscription_identifier as std::string, when it's actually std::vector<std::string>. Same applies for user_property. - As of Boost 1.86, timeouts should be implemented using asio::cancel_after, rather than by awaitable operators or parallel groups. I suggest replacing the examples on timeouts by a single one using asio::cancel_after. - Type requirements for ExecutionContext (https://spacetime.mireo.com/async-mqtt5/async_mqtt5/ref/ExecutionContext.ht…) are not correct. You can probably just link to Asio's description: https://www.boost.org/doc/libs/1_86_0/doc/html/boost_asio/reference/Executi…. - The multithreaded example recommends the following code to run callbacks though a strand: boost::asio::post( strand, [&client, &strand] { // Considering that the default associated executor of all completion handlers is the strand, // it is not necessary to explicitly bind it to async_run or other async_xxx's handlers. client.async_run(boost::asio::detached); } ); This form of post is not equivalent to asio::post(asio::bind_executor(ex, ...)), and can yield surprising behavior if the token has a bound executor. Furthermore, post might be less efficient than dispatch, and should only be used if a reschedule is required (which is not the case). I'd advise to change the above by: boost::asio::dispatch(boost::asio::bind_executor( strand, [&client, &strand] { // Considering that the default associated executor of all completion handlers is the strand, // it is not necessary to explicitly bind it to async_run or other async_xxx's handlers. client.async_run(boost::asio::detached); } )); - The multi-threaded example uses asio::io_context and a vector of threads. This is error-prone. It can be simplified by using asio::thread_pool, asio::thread_pool::attach and asio::thread_pool::join. - The coroutine examples can be simplified with the latest Asio additions. At the moment, asio::deferred is recommended over asio::use_awaitable for efficiency reasons. With partial tokens, async_xxx(asio::as_tuple(asio::deferred)) can be replaced by async_xxx(asio::as_tuple). - The coroutines examples shouldn't use asio::detached with asio::co_spawn, as this swallows any thrown exceptions. It's better to use [](std::exception_ptr e) { if(e) std::rethrow_exception(e); }, as this propagates exceptions to main. - It's probably better to avoid the unified header in the examples. We should encourage including individual headers, instead, as this is better for compile times. - In https://spacetime.mireo.com/async-mqtt5/async_mqtt5/connection_maintenance.…: should be "the MQTT <protocol>" (missing word). - https://spacetime.mireo.com/async-mqtt5/async_mqtt5/optimising_communicatio…: typo: "simultanious". - https://spacetime.mireo.com/async-mqtt5/async_mqtt5/ref/mqtt_client/operato…: moved-from state probably supports being assigned to, too. - https://spacetime.mireo.com/async-mqtt5/async_mqtt5/ref/mqtt_client/tls_con…: render error in the 1st template parameter. - https://spacetime.mireo.com/async-mqtt5/async_mqtt5/ref/mqtt_client/authent…: render error in the 1st template parameter. - https://spacetime.mireo.com/async-mqtt5/async_mqtt5/ref/authority_path.html: member descriptions are empty. Thanks Ivica for proposing the library. I really hope to see it in Boost in the near future. Regards, Ruben.

2 2

Re: [boost] [async-mqtt5] Review questions
by Ruben Perez 18 Oct '24

18 Oct '24

(CC-ing the list) > > > > I haven't looked at the implementation yet, but I'm assuming that > > owning types are being used for a reason. > > You mean like ASIO buffers? Or vector? To std::move them around? I mean that the current interface is using std::string (owning) as the argument type for message payloads, which probably means that the implementation will std::move the supplied buffer, so the user doesn't need to keep it alive. My question was regarding the use of a character type (like std::string) vs a blob type (like std::vector<std::byte>), more than an owning vs. a non-woning type. Now that you've mentioned it, most Asio-based libraries require the user to keep the message payload alive for the duration of the operation, in an attempt to minimize copying. So I will add this to my list of questions for Ivica: is there a reason why you decided to use an owning type for the message payload? Regards, Ruben.

1 0

[release] Boost 1.87.0 closes for new libraries on 23-October
by Marshall Clow 16 Oct '24

16 Oct '24

As always, the calendar is at: https://www.boost.org/development/index.html — The release managers

1 0

Boost cross-compilation and testing (Linux -> Windows)
by Martin Vejbora 16 Oct '24

16 Oct '24

Hello, we're working on a new target for MinGW compiler, we're intending to have a cross-compiler from Linux Arm64 to Windows Arm64. We want to have Boost library as a benchmark of how complete our C++ compiler is and gradually improve it to support more features. I'm struggling though to get the cross-compilation and subsequent testing working. 1) I'm having troubles to set up cross-building of Boost with MinGW (even standard MinGW from package manager Linux x64 -> Windows x64). I'm getting errors while building Context, Process and Stacktrace modules. (I added to the respective error messages to the end of the email or full logs can be found here: https://github.com/user-attachments/files/17133738/x64_boost_build.log) My build commands: git clone --recursive https://github.com/boostorg/boost.git boost-build cd boost-build git checkout develop git submodule update --init --recursive echo "using gcc : : x86_64-w64-mingw32-g++ ;" > "user-config-x64.jam" ./bootstrap.sh ./b2 --user-config=./user-config-x64.jam --prefix=./boost-x64 --debug-configuration target-os=windows address-model=64 variant=debug install > x64_boost_build.log >From the error messages, it seems like I'm doing something essential wrong as for example Process module tries to build assembler for ELF format and SYSV ABI and not Window's PE format and MS ABI. 2) Subsequently, I didn't have much luck finding resources for cross-testing of Boost. Currently, I'm building and running tests in Ubuntu 22.04 in WSL on Windows machine. In general, this approach seems to work decently, Boost itself and its tests are built on Ubuntu as Windows binaries and when test executables are run, WSL automatically propagates it to run those executables on underlaying Windows. But a lot of tests are failing. I'm gradually investigating them and finding issues like missing dlls because b2 uses LD_LIBRARY_PATH and Windows doesn't understand it. I'm workarounding these issues but it's a slow process, so I wanted to ask, is the cross-testing of Boost library supported, are you aware of some examples where it's being used? Thank you and kind regards, Martin ------------------------------------------------------------------------------- a) Context errors: (similar also for jump_x86_64_sysv_elf_gas.S and ontop_x86_64_sysv_elf_gas.S) gcc.compile.asm bin.v2/libs/context/build/gcc-10/debug/x86_64/link-static/ target-os-windows/threadapi-win32/threading-multi/visibility-hidden/ asm/make_x86_64_sysv_elf_gas.o make_x86_64_sysv_elf_gas.S: Assembler messages: make_x86_64_sysv_elf_gas.S:47: Warning: .type pseudo-op used outside of .def/.endef: ignored. make_x86_64_sysv_elf_gas.S:47: Error: junk at end of line, first unrecognized character is `m' /home/vejby/boost-build/libs/context/src/asm/make_x86_64_sysv_elf_gas.S:157: Warning: .size pseudo-op used outside of .def/.endef: ignored. /home/vejby/boost-build/libs/context/src/asm/make_x86_64_sysv_elf_gas.S:157: Error: junk at end of line, first unrecognized character is `m' /home/vejby/boost-build/libs/context/src/asm/make_x86_64_sysv_elf_gas.S:160: Error: junk at end of line, first unrecognized character is `-' "x86_64-w64-mingw32-g++" -x assembler-with-cpp -m64 -O0 -fno-inline -Wall -g -fvisibility=hidden -DBOOST_ALL_NO_LIB=1 -DBOOST_CONTEXT_NO_LIB=1 -DBOOST_CONTEXT_SOURCE -D_WIN32_WINNT=0x0601 -I"." -I"/home/vejby/boost-build/libs/assert/include" -I"/home/vejby/boost-build/libs/context/include" -I"/home/vejby/boost-build/libs/core/include" -I"/home/vejby/boost-build/libs/move/include" -I"/home/vejby/boost-build/libs/mp11/include" -I"/home/vejby/boost-build/libs/predef/include" -I"/home/vejby/boost-build/libs/smart_ptr/include" -I"/home/vejby/boost-build/libs/throw_exception/include" -I"/home/vejby/boost-build/libs/type_traits/include" -c -o "bin.v2/libs/context/build/gcc-10/debug/x86_64/link-static/target-os-windows/ threadapi-win32/threading-multi/visibility-hidden/asm/make_x86_64_sysv_elf_gas.o" "/home/vejby/boost-build/libs/context/src/asm/make_x86_64_sysv_elf_gas.S" ...failed gcc.compile.asm bin.v2/libs/context/build/gcc-10/debug/x86_64/link-static/target-os-windows/ threadapi-win32/threading-multi/visibility-hidden/asm/make_x86_64_sysv_elf_gas.o... ------------------------------------------------------------------------------- b) Process errors: gcc.compile.c++ bin.v2/libs/process/build/gcc-10/debug/x86_64/ boost.process.fs-boost/link-static/target-os-windows/threadapi-win32/ threading-multi/visibility-hidden/ext/exe.o /home/vejby/boost-build/libs/process/src/ext/exe.cpp: In function ‘boost::filesystem::path boost::process::v2::ext::exe (HANDLE, boost::system::error_code&)’: /home/vejby/boost-build/libs/process/src/ext/exe.cpp:78:9: error: ‘QueryFullProcessImageNameW’ was not declared in this scope 78 | if (QueryFullProcessImageNameW(proc, 0, buffer, &size)) | ^~~~~~~~~~~~~~~~~~~~~~~~~~ "x86_64-w64-mingw32-g++" -fvisibility-inlines-hidden -m64 -O0 -fno-inline -Wall -g -fvisibility=hidden -DBOOST_ALL_NO_LIB=1 -DBOOST_ASIO_NO_DEPRECATED -DBOOST_ATOMIC_STATIC_LINK=1 -DBOOST_FILESYSTEM_NO_LIB=1 -DBOOST_FILESYSTEM_STATIC_LINK=1 -DBOOST_PROCESS_SOURCE=1 -DBOOST_REGEX_NO_LIB=1 -DBOOST_SYSTEM_NO_LIB=1 -DBOOST_SYSTEM_STATIC_LINK=1 -DWIN32_LEAN_AND_MEAN -I"." -I"/home/vejby/boost-build/libs/algorithm/include" -I"/home/vejby/boost-build/libs/assert/include" -I"/home/vejby/boost-build/libs/bind/include" -I"/home/vejby/boost-build/libs/container_hash/include" -I"/home/vejby/boost-build/libs/core/include" -I"/home/vejby/boost-build/libs/describe/include" -I"/home/vejby/boost-build/libs/detail/include" -I"/home/vejby/boost-build/libs/filesystem/include" -I"/home/vejby/boost-build/libs/function/include" -I"/home/vejby/boost-build/libs/function_types/include" -I"/home/vejby/boost-build/libs/fusion/include" -I"/home/vejby/boost-build/libs/iterator/include" -I"/home/vejby/boost-build/libs/move/include" -I"/home/vejby/boost-build/libs/mp11/include" -I"/home/vejby/boost-build/libs/optional/include" -I"/home/vejby/boost-build/libs/predef/include" -I"/home/vejby/boost-build/libs/process/include" -I"/home/vejby/boost-build/libs/regex/include" -I"/home/vejby/boost-build/libs/scope/include" -I"/home/vejby/boost-build/libs/smart_ptr/include" -I"/home/vejby/boost-build/libs/system/include" -I"/home/vejby/boost-build/libs/throw_exception/include" -I"/home/vejby/boost-build/libs/tuple/include" -I"/home/vejby/boost-build/libs/type_index/include" -I"/home/vejby/boost-build/libs/type_traits/include" -I"/home/vejby/boost-build/libs/unordered/include" -I"/home/vejby/boost-build/libs/variant2/include" -I"/home/vejby/boost-build/libs/winapi/include" -c -o "bin.v2/libs/process/build/gcc-10/debug/x86_64/boost.process.fs-boost/ link-static/target-os-windows/threadapi-win32/threading-multi/ visibility-hidden/ext/exe.o" "/home/vejby/boost-build/libs/process/src/ext/exe.cpp" ...failed gcc.compile.c++ bin.v2/libs/process/build/gcc-10/debug/x86_64/boost.process.fs-boost/ link-static/target-os-windows/threadapi-win32/threading-multi/ visibility-hidden/ext/exe.o... ------------------------------------------------------------------------------- gcc.compile.c++ bin.v2/libs/process/build/gcc-10/debug/x86_64/ boost.process.fs-boost/link-static/target-os-windows/threadapi-win32/ threading-multi/visibility-hidden/windows/default_launcher.o In file included from /home/vejby/boost-build/libs/process/src/windows/ default_launcher.cpp:16: ./boost/process/v2/windows/default_launcher.hpp:220:3: error: ‘STARTUPINFOEXW’ does not name a type; did you mean ‘STARTUPINFOW’? 220 | STARTUPINFOEXW startup_info{{sizeof(STARTUPINFOEXW), nullptr, nullptr, nullptr, | ^~~~~~~~~~~~~~ | STARTUPINFOW ./boost/process/v2/windows/default_launcher.hpp: In member function ‘boost::process::v2::windows::default_launcher::enable_init <Executor, Inits ...> boost::process::v2::windows::default_launcher:: operator()(Executor, boost::system::error_code&, const typename std::enable_if<( boost::asio::execution::is_executor<T>::value || boost::asio::is_executor<Executor>::value ), boost::filesystem::path>::type&, Args&&, Inits&& ...)’: ./boost/process/v2/windows/default_launcher.hpp:312:10: error: ‘startup_info’ was not declared in this scope 312 | &startup_info.StartupInfo, | ^~~~~~~~~~~~ /home/vejby/boost-build/libs/process/src/windows/default_launcher.cpp: In static member function ‘static std::size_t boost::process::v2::windows:: default_launcher::escape_argv_string(wchar_t*, std::size_t, boost::basic_string_view<wchar_t, std::char_traits<wchar_t> >)’: /home/vejby/boost-build/libs/process/src/windows/default_launcher.cpp:64:16: warning: unused variable ‘end’ [-Wunused-variable] 64 | const auto end = itr + sz; | ^~~ "x86_64-w64-mingw32-g++" -fvisibility-inlines-hidden -m64 -O0 -fno-inline -Wall -g -fvisibility=hidden -DBOOST_ALL_NO_LIB=1 -DBOOST_ASIO_NO_DEPRECATED -DBOOST_ATOMIC_STATIC_LINK=1 -DBOOST_FILESYSTEM_NO_LIB=1 -DBOOST_FILESYSTEM_STATIC_LINK=1 -DBOOST_PROCESS_SOURCE=1 -DBOOST_REGEX_NO_LIB=1 -DBOOST_SYSTEM_NO_LIB=1 -DBOOST_SYSTEM_STATIC_LINK=1 -DWIN32_LEAN_AND_MEAN -I"." -I"/home/vejby/boost-build/libs/algorithm/include" -I"/home/vejby/boost-build/libs/assert/include" -I"/home/vejby/boost-build/libs/bind/include" -I"/home/vejby/boost-build/libs/container_hash/include" -I"/home/vejby/boost-build/libs/core/include" -I"/home/vejby/boost-build/libs/describe/include" -I"/home/vejby/boost-build/libs/detail/include" -I"/home/vejby/boost-build/libs/filesystem/include" -I"/home/vejby/boost-build/libs/function/include" -I"/home/vejby/boost-build/libs/function_types/include" -I"/home/vejby/boost-build/libs/fusion/include" -I"/home/vejby/boost-build/libs/iterator/include" -I"/home/vejby/boost-build/libs/move/include" -I"/home/vejby/boost-build/libs/mp11/include" -I"/home/vejby/boost-build/libs/optional/include" -I"/home/vejby/boost-build/libs/predef/include" -I"/home/vejby/boost-build/libs/process/include" -I"/home/vejby/boost-build/libs/regex/include" -I"/home/vejby/boost-build/libs/scope/include" -I"/home/vejby/boost-build/libs/smart_ptr/include" -I"/home/vejby/boost-build/libs/system/include" -I"/home/vejby/boost-build/libs/throw_exception/include" -I"/home/vejby/boost-build/libs/tuple/include" -I"/home/vejby/boost-build/libs/type_index/include" -I"/home/vejby/boost-build/libs/type_traits/include" -I"/home/vejby/boost-build/libs/unordered/include" -I"/home/vejby/boost-build/libs/variant2/include" -I"/home/vejby/boost-build/libs/winapi/include" -c -o "bin.v2/libs/process/build/gcc-10/debug/x86_64/boost.process.fs-boost/ link-static/target-os-windows/threadapi-win32/threading-multi/ visibility-hidden/windows/default_launcher.o" "/home/vejby/boost-build/libs/process/src/windows/default_launcher.cpp" ...failed gcc.compile.c++ bin.v2/libs/process/build/gcc-10/debug/x86_64/boost.process.fs-boost/ link-static/target-os-windows/threadapi-win32/threading-multi/ visibility-hidden/windows/default_launcher.o... ------------------------------------------------------------------------------- gcc.compile.c++ bin.v2/libs/process/build/gcc-10/debug/x86_64 boost.process.fs-boost/link-static/target-os-windows/threadapi-win32/ threading-multi/visibility-hidden/environment.o In file included from ./boost/process/v2/default_launcher.hpp:17, from /home/vejby/boost-build/libs/process/src/environment.cpp:13: ./boost/process/v2/windows/default_launcher.hpp:220:3: error: ‘STARTUPINFOEXW’ does not name a type; did you mean ‘STARTUPINFOW’? 220 | STARTUPINFOEXW startup_info{{sizeof(STARTUPINFOEXW), nullptr, nullptr, nullptr, | ^~~~~~~~~~~~~~ | STARTUPINFOW ./boost/process/v2/windows/default_launcher.hpp: In member function ‘boost::process::v2::windows::default_launcher::enable_init<Executor, Inits ...> boost::process::v2::windows::default_launcher::operator()(Executor, boost::system::error_code&, const typename std::enable_if<( boost::asio::execution::is_executor<T>::value || boost::asio::is_executor<Executor>::value ), boost::filesystem::path>::type&, Args&&, Inits&& ...)’: ./boost/process/v2/windows/default_launcher.hpp:312:10: error: ‘startup_info’ was not declared in this scope 312 | &startup_info.StartupInfo, | ^~~~~~~~~~~~ "x86_64-w64-mingw32-g++" -fvisibility-inlines-hidden -m64 -O0 -fno-inline -Wall -g -fvisibility=hidden -DBOOST_ALL_NO_LIB=1 -DBOOST_ASIO_NO_DEPRECATED -DBOOST_ATOMIC_STATIC_LINK=1 -DBOOST_FILESYSTEM_NO_LIB=1 -DBOOST_FILESYSTEM_STATIC_LINK=1 -DBOOST_PROCESS_SOURCE=1 -DBOOST_REGEX_NO_LIB=1 -DBOOST_SYSTEM_NO_LIB=1 -DBOOST_SYSTEM_STATIC_LINK=1 -DWIN32_LEAN_AND_MEAN -I"." -I"/home/vejby/boost-build/libs/algorithm/include" -I"/home/vejby/boost-build/libs/assert/include" -I"/home/vejby/boost-build/libs/bind/include" -I"/home/vejby/boost-build/libs/container_hash/include" -I"/home/vejby/boost-build/libs/core/include" -I"/home/vejby/boost-build/libs/describe/include" -I"/home/vejby/boost-build/libs/detail/include" -I"/home/vejby/boost-build/libs/filesystem/include" -I"/home/vejby/boost-build/libs/function/include" -I"/home/vejby/boost-build/libs/function_types/include" -I"/home/vejby/boost-build/libs/fusion/include" -I"/home/vejby/boost-build/libs/iterator/include" -I"/home/vejby/boost-build/libs/move/include" -I"/home/vejby/boost-build/libs/mp11/include" -I"/home/vejby/boost-build/libs/optional/include" -I"/home/vejby/boost-build/libs/predef/include" -I"/home/vejby/boost-build/libs/process/include" -I"/home/vejby/boost-build/libs/regex/include" -I"/home/vejby/boost-build/libs/scope/include" -I"/home/vejby/boost-build/libs/smart_ptr/include" -I"/home/vejby/boost-build/libs/system/include" -I"/home/vejby/boost-build/libs/throw_exception/include" -I"/home/vejby/boost-build/libs/tuple/include" -I"/home/vejby/boost-build/libs/type_index/include" -I"/home/vejby/boost-build/libs/type_traits/include" -I"/home/vejby/boost-build/libs/unordered/include" -I"/home/vejby/boost-build/libs/variant2/include" -I"/home/vejby/boost-build/libs/winapi/include" -c -o "bin.v2/libs/process/build/gcc-10/debug/x86_64/boost.process.fs-boost/ link-static/target-os-windows/threadapi-win32/threading-multi/ visibility-hidden/environment.o" "/home/vejby/boost-build/libs/process/src/environment.cpp" ...failed gcc.compile.c++ bin.v2/libs/process/build/gcc-10/debug/x86_64/boost.process.fs-boost/ link-static/target-os-windows/threadapi-win32/threading-multi/ visibility-hidden/environment.o... ------------------------------------------------------------------------------- c) Stacktrace error: gcc.compile.c++ bin.v2/libs/stacktrace/build/gcc-10/debug/x86_64/link-static/ target-os-windows/threadapi-win32/threading-multi/visibility-hidden/ from_exception.o /home/vejby/boost-build/libs/stacktrace/build/../src/from_exception.cpp:161:10: fatal error: dlfcn.h: No such file or directory 161 | #include <dlfcn.h> | ^~~~~~~~~ compilation terminated. "x86_64-w64-mingw32-g++" -fvisibility-inlines-hidden -m64 -O0 -fno-inline -Wall -g -fvisibility=hidden -DBOOST_ALL_NO_LIB=1 -I"." -I"/home/vejby/boost-build/libs/assert/include" -I"/home/vejby/boost-build/libs/container_hash/include" -I"/home/vejby/boost-build/libs/core/include" -I"/home/vejby/boost-build/libs/describe/include" -I"/home/vejby/boost-build/libs/mp11/include" -I"/home/vejby/boost-build/libs/predef/include" -I"/home/vejby/boost-build/libs/stacktrace/include" -I"/home/vejby/boost-build/libs/throw_exception/include" -I"/home/vejby/boost-build/libs/winapi/include" -c -o "bin.v2/libs/stacktrace/build/gcc-10/debug/x86_64/link-static/ target-os-windows/threadapi-win32/threading-multi/visibility-hidden/ from_exception.o" "/home/vejby/boost-build/libs/stacktrace/build/../src/from_exception.cpp" ...failed gcc.compile.c++ bin.v2/libs/stacktrace/build/gcc-10/debug/x86_64/link-static/target-os-windows/ threadapi-win32/threading-multi/visibility-hidden/from_exception.o...

3 4