- Boost - lists.boost.org

need help with building QuickBook docs
by Andrzej Krzemienski 23 May '22

23 May '22

Hi Everyone, This is to ask for help with building QuickBook documentation, but also to spill my frustration. I would like to offer some feedback in connection with the recent "the future of Boost" threads: about the entry barrier for someone who wants to contribute to Boost. I maintain Boost.Optional. Getting my environment to the point where I was actually able to apply modifications to the official repo took some time, but it has been working fine since. Modifying the source files, so that they compile on multiple compilers, platforms and versions, including C++03 is the easiest part. Testing is a bit harder, but basically works also: I go to folder `test`, run the `b2` command, and it is done. But updating the documentation is the most difficult task for me. Not the content -- this is easy -- but the process/pipeline/framework. I inherited the documentation in QuickBook format, which as I understand was the recommended way for providing documentation for Boost libraries. Maybe it still is? (but I see more and more docs being rewritten to AsciDocs.) I understand that library authors have the freedom to choose their own format of docs for their libraries, but does boost recommend (as opposed to enforcing) one format? Anyway, in order to test if my QuickBook dos are correct, I need to build HTML version from the sources. Sometimes it "just works" with `b2` command, but when something goes wrong, this becomes a pain. I cannot just call `b2` for my library, because apparently my Fedora comes with preinstalled version of b2 (4.4) which appears to be incompatible with something in Boost libraries sources, as I get error: error: mismatched versions of B2 engine and core > error: B2 engine (b2) is 4.4.1 > error: B2 core (at /home/andrzej/Repos/boost/tools/build/src) is 4.9-git > Ok, I learned to live with it and to launch b2 from tools/build. It used to work for a while, but yesterday when I did a `git pull --rebase` on Boost repo, it no longer works. I get an error: Jamfile.v2:45: Unescaped special character in argument > <format>pdf:<xsl:param>boost.url.prefix= > http://www.boost.org/doc/libs/release/libs/optional/doc/html > Performing configuration checks > > - default address-model : 64-bit (cached) [1] > error: at /home/andrzej/Repos/boost/tools/build/src/kernel/modules.jam:106 > error: Unable to find file or target named > error: '/boost/architecture//mips' > error: referred to from project at > error: '.' > Ok, I can remove line `: /boost/architecture//mips` from file `boostcpp.jam`. (Am I supposed to do it before using `b2`?). But then, when trying to build the docs, I get a lot of errors, saying: runtime error: file > /home/andrzej/Repos/boost/tools/boostbook/xsl/annotation.xsl line 432 > element element > xsl:element: The effective name '' is not a valid QName. > (Full error message is enclosed.) I have no clue which element in the pipeline is causing or reporting this. QuickBook, BoostBook, DocBook, xsltproc? The process involves many components, at least according to this diagram: https://svn.boost.org/trac10/wiki/BoostDocs/GettingStarted Is this something obvious that I am missing? Is this a good idea, or a recommended practice to use QuickBook nowadays for Boost docs? I feel comfortable fixing or writing C++ code, and I am willing to devote some of my limited time to do this. But fighting with QuickBook documentation is something I would not rather do. I have a small library that I considered submitting to Boost: https://github.com/akrzemi1/markable But as long as it is on my GitHub page, it is simple to maintain, test and document. If I imagine how many nights I will have to spend making the documentation compatible with Boost, or getting all the b2-related stuff to work, I feel discouraged. Maybe I am being unfair to `b2` now. But it is certainly my personal emotional impression. I do not mind the current Boost website format or the underlying technology. I do not mind the mailing list format. My primary obstacle is this perceived or real difficulty to get a boost library into a shape where it just builds, including its documentation. My ideal as a Boost library maintainer, is that I just provide the source for the library and the documentation, and anything else "just works". Maybe this is not achievable, but the closer to this ideal, the more encouraged I would be to consider authoring or maintaining a library. Any help in fixing my build problem is most welcome. Regards, &rzej;

7 16

MySql review
by Phil Endecott 23 May '22

23 May '22

Here is my review of Ruben Perez's proposed MySql library. Background ---------- I have previously implemented C++ wrappers for PostgreSQL and SQLite, so I have some experience of what an SQL API can look like. I know little about ASIO. I have also recently used the AWS SDKs for C++ and Javascript to talk to DynamoDB; this has async functionality, which is interesting to compare. I confess some minor disappointment that MySql, rather than PostgreSQL or SQLite, is the subject of this first Boost database library review, since those others have liberal licences that are closer to Boost's own licence than MySql (and MariaDB). But I don't think that should be a factor in the review. Trying the library ------------------ I have tried using the library with - g++ 10.2.1, Arm64, Debian Linux - ASIO from Boost 1.74 (Debian packages) - Amazon Aurora MySql-compatible edition I've written a handful of simple test programs. Everything works as expected. Compilation times are a bit slow but not terrible. The remainder of this review approximately follows the structure of the library documentation. Introduction ------------ I note that "Ease of use" is claimed as the first design goal, which is good. I feel that some mention should be made of the existing C / C++ APIs and their deficiencies. You should also indicate whether or not the network protocol you are using to communicate with the server is a "public" interface with some sort of stability guarantee. (I guess maybe it is, if it is common to MySql and MariaDB.) Tutorial -------- The code fragments should start with the necessary #includes, OR you should prominently link to the complete tutorial source code at the start. You say that "this tutorial assumes you have a basic familiarity with Boost.Asio". I think that's unfortunate. It should be possible for someone to use much of the library's functionality knowing almost nothing about ASIO. Remember your design goal of ease-of-use. In fact, it IS possible to follow the tutorial with almost no knowledge of ASIO because I have just done so. You have this boilerplate at the start of the tutorial: boost::asio::io_context ctx; boost::asio::ssl::context ssl_ctx (boost::asio::ssl::context::tls_client); boost::mysql::tcp_ssl_connection conn (ctx.get_executor(), ssl_ctx); boost::asio::ip::tcp::resolver resolver (ctx.get_executor()); auto endpoints = resolver.resolve(argv[3], boost::mysql::default_port_string); boost::mysql::connection_params params ( argv[1], // username argv[2] // password ); conn.connect(*endpoints.begin(), params); // I guess that should really be doing something more // intelligent than just trying the first endpoint, right? I would like to see a convenience function that hides all of that: auto conn = boost::mysql::make_connection( ...params... ); I guess this will need to manage a global, private, ctx object or something. There are various possibilities for the connection parameters to pass to that, e.g. connection_params params = { .hostname = ".....", .port = 3306, // why is that a string in yours? .user = "admin", .password = "12345", .dbname = "foo" }; make_connection(params); Or make_connection("mysql://admin:12345@hostname:3306/dbname"); Now... why the heck does your connection_params struct use string_views? That ought to be a Regular Type, with Value Semantics, using std::strings. Is this the cult of not using strings because "avoid copying above all else"? Another point about the connection parameters: you should provide a way to supply credentials without embedding them in the source code. You should aim to make the secure option the default and the simplest to use. I suggest that you support the ~/.my.cnf and /etc/my.cnf files and read passwords etc. from there, by default. You might also support getting credentials from environment variables or by parsing the command line. You could even have it prompt for a password. Does MySQL support authentication using SSL client certs? I try to use this for PostgreSQL when I can. If it does, you should try to support that too. About two thirds of the way through the tutorial, it goes from "Hello World" to retrieving "employees". Please finish the hello world example with code that gets the "Hello World" string from the results and prints it. Queries ------- I encourage you to present prepared queries first in the documentation and to use them almost exclusively in the tutorial and examples. You say that "client side query composition is not available". What do you mean by "query composition"? I think you mean concatenating strings together'); drop table users; -- to form queries, right? Is that standard MySql terminology? I suggest that you replace the term with something like "dangerous string concatenation". In any case, that functionality *is* available, isn't it! It's trivial to concatenate strings and pass them to your text query functions. You're not doing anything to block that. So what you're really saying is that you have not provided any features to help users do this *safely*. I think that's a serious omission. It would not be difficult for you to provide an escape_for_mysql_quoted_string() function, rather than having every user roll their own slightly broken version. IIRC, in PostgreSQL you can only use prepared statements for SELECT, UPDATE, INSERT and DELETE statements; if you want to do something like ALTER TABLE a ALTER COLUMN c SET DEFAULT = ? or CREATE VIEW v as SELECT * FROM t WHERE c = ? then you are forced to do string concatenation with escaping of parameters. But according to https://dev.mysql.com/doc/refman/8.0/en/sql-prepared-statements.html, MySQL allows more statements to be prepared. In what cases can prepared statements not be used? Perhaps non-prepared statements really can be hidden as mysql::unsafe::statement? Regarding your prepared statements: tcp_ssl_prepared_statement is verbose. Why does the prepared statement type depend on the underlying connection type? I have to change it if I change the connection type?! If that's unavoidable, I suggest putting a type alias in the connection type: connection_t conn = ....; connection_t::prepared_statement stmt(.....); Does MySql allow numbered or named parameters? SQLite supports ?1 and :name; I think PostgreSQL uses $n. Queries with lots of parameters are error-prone if you just have ?. If MySql does support this, it would be good to see it used in some of the examples. Invoking the prepared statement seems unnecessarily verbose. Why can't I just write auto result = my_query("hello", "world", 42); ? That's a variadic operator(). In my PostgreSQL library, I chose to specify the query parameter types as template parameters to the query (prepared statement) type: Query<std::string, std::string, int> my_query("....."); Query's operator() takes exactly those types (maybe with some TMP to use std::string_view where std::string is specified). With hindsight, a default where all parameters are strings would have been useful in practice. I also added Query variants where the result is expected to be - A single value, e.g. a SELECT COUNT(*) statement. - Empty, e.g. INSERT or DELETE. - A single row. - Zero or one rows. - A single column. This is just syntactic sugar but it makes life easier. For example: SingletonQuery<int, string> sessions_for_user("SELECT COUNT(*) FROM sessions WHERE username = ?"); // int is the return type, string is the one parameter type. auto n_sessions = sessions_for_user("phil"); I don't see anything about the result of an INSERT or UPDATE. PostgreSQL tells me the number of rows affected, which I have found useful to return to the application. resultset, row and value ------------------------ I'm not enthusiastic about the behaviour nor the names of these types: - resultset is not a set of results. It's more like a sequence of rows. But more importantly, it's lazy; it's something like an input stream, or an input range. So why not actually make it an input range, i.e. make it "a model of the input_range concept". Then we could write things like: auto result = ...execute query... for (auto&& row: result) { ... } - row: not bad, it does actually represent a row; it's a shame it's not a regular type though. - value: it's not a value! It doesn't have value semantics! I guess it is time to address the string_view issue..... I just modified one of my example programs to copy from the string_view in the variant into a std::string, and it slowed the program down by... no, it actually went fractionally faster. The use of string_views in the value variant seems to me to be a premature optimisation. There are significant disadvantages to using string_view here. Your default API should have value semantics. If you want, provide another "low copying" API for users who want it with names that make it clear what they are doing. The problem is that the lifetime of the string_view is tied to the row in a non-obvious way. I'm also uncertain that a variant for the individual values is the right solution here. All the values in a column should have the same type, right? (Though some can be null.) So I would make row a tuple. Rather than querying individual values for their type, have users query the column. MySQL to C++ mapping reference ------------------------------ It seems odd that MySQL small integers all map to C++ 64-bit types. I use NUMERIC quite a lot in PostgreSQL; I don't know if the MySql type is similar. I would find it inconvenient that it is treated as a string. Can it not be converted to a numeric type, if that is what the user wants? Need to consume the entire resultset ------------------------------------ I seem to get an assertion if I fail to read the resultset (resultset.hpp:70). Could this throw instead? Or, should the library read and discard the unread results in this case? Going Async ----------- I have tried this with both callbacks and C++20 coroutines and it seems to work as expected. I'm actually delighted to see the support for coroutines here. But the lack of protocol support for multiple in-flight queries immediately becomes apparent. It almost makes me question the value of the library - what's the point of the async support, if we then have to serialise the queries? Should the library provide this serialisation for us? I.e. if I async_execute a query while another is in progress, the library could wait for the first to complete before starting the second. Or, should the library provide a connection pool? (Does some other part of ASIO provide connection pool functionality that can be used here?) Transactions ------------ I have found it useful to have a Transaction class: { Transaction t(conn); // Issues "BEGIN" .... run queries .... t.commit(); // Issues "COMMIT" } // t's dtor issues "ROLLBACK" if we have not committed. Nested transactions are an interesting problem. Trademark --------- Klemens Morgenstern makes the point that MySql is a trademark of Oracle. Calling this "Boost.MySql" doesn't look great to me. How can you write "The Boost MySql-compatible Database Library" more concisely? Conclusion ========== Good points: It actually works! Great to see coroutines! Bad points: The resultset / row / value type system is a poor design. A lot of boilerplate could be hidden from many users. Connection pool may be required for useful performance. Several features should be re-worked to be secure by default. Use of MySQL trademark needs to be considered. Overall, I think this proposal needs a fair amount of API re-design and additional features to be accepted, and should be rejected at this time. It does seem to be a good start though! Thanks to Ruben for the submission. (I have spend one working day on this review. I have not looked at any of the code.) Regards, Phil.

13 18

Re: [boost] need help with building QuickBook docs
by Andrzej Krzemienski 22 May '22

22 May '22

niedz., 22 maj 2022 o 17:33 <pbristow(a)hetp.u-net.com> napisał(a): > I don’t see where annotation.xsl comes from, or what is should contain > (but it looks as though it contains something wrong!!). > > > > Can you share it with us? And any metadata? > The file comes from the boostbook repo: https://github.com/boostorg/boostbook/blob/develop/xsl/annotation.xsl I applied no modifications. > > > None of this looks familiar to me from generating Quickbook docs ☹ > > > > Paul > > > > PS I also view items reported as cached with suspicion, as a result of > being bitten before. > > > > Especially when there are two difference versions. > > > > > https://stackoverflow.com/questions/5709105/how-do-i-force-bjam-in-boost-to… > may help. > Thanks for the idea. I'll try it later, in the evening. Regards, &rzej; > > > > > *From:* Andrzej Krzemienski <akrzemi1(a)gmail.com> > *Sent:* 22 May 2022 15:45 > *To:* Boost mailing list <boost(a)lists.boost.org> > *Cc:* pbristow(a)hetp.u-net.com > *Subject:* Re: [boost] need help with building QuickBook docs > > > > > > > > niedz., 22 maj 2022 o 15:52 Paul A Bristow via Boost < > boost(a)lists.boost.org> napisał(a): > > > -----Original Message----- > > From: Boost <boost-bounces(a)lists.boost.org> On Behalf Of Andrzej > Krzemienski via Boost > > Sent: 22 May 2022 14:14 > > To: René Ferdinand Rivera Morell <grafikrobot(a)gmail.com> > > Cc: Andrzej Krzemienski <akrzemi1(a)gmail.com>; Boost Developers List < > boost(a)lists.boost.org> > > Subject: Re: [boost] need help with building QuickBook docs > > > > Thanks. That would be the most likely cause of my problems. Could you > point me to instructions for > > rebuilding the b2 engine executable? > > > > https://www.boost.org/doc/libs/1_79_0/more/getting_started/windows.html#get… > > > Section 5.1 Simplified Build From Source > > If you wish to build from source with Visual C++, you can use a simple > build procedure described in this section. Open the command prompt and > change your current directory to the Boost root directory. Then, type the > following commands: > > bootstrap > .\b2 > The first command prepares the Boost.Build system for use. The second > command invokes Boost.Build to build the separately-compiled Boost > libraries. Please consult the Boost.Build documentation for a list of > allowed options. > > https://www.boost.org/doc/libs/1_79_0/tools/build/doc/html/index.html > > > https://www.boost.org/doc/libs/1_79_0/tools/build/doc/html/index.html#bbv2.… > > Run either .\bootstrap.bat (on Windows), or ./bootstrap.sh (on other > operating systems) > > (which may be too simple. By default, Windows 10 and up, probably don't > provide cmd.exe, preferring the fancy new command shell Windows > PowerShell) > > b2 -v will show the current version. > > You will probably want to ensure that b2.exe is visible globally, perhaps > by adding to PATH. Perhaps using the Windows Environment Variables editor? > > HTH > > Paul > > > > Thank you. This seems to remove the first obstacle on my way to building > Boost.Optional docs. > > I am now using B2 Version 4.9. OS=LINUX. I get the error from another > tool, I suppose: > > > > [2] > gcc-12/release/cxxstd-0x-iso/link-static/python-3.10/threadapi-pthread/threading-multi/visibility-hidden > ...patience... > ...patience... > ...found 4351 targets... > ...updating 2 targets... > xslt-xsltproc ../../../bin.v2/libs/optional/doc/optional.docbook > runtime error: file > /home/andrzej/Repos/boost/tools/boostbook/xsl/annotation.xsl line 432 > element element > xsl:element: The effective name '' is not a valid QName. > > > > Is it DocBook, QuickBok, BoostBook? (I enclose the full error report.) I > am just running b2 on boost/libs/optional/doc from the official repo. Can > someone confirm if they see the same problem? or is it just me? > > > > Regards, > > &rzej; > > > > > > > > > > > _______________________________________________ > Unsubscribe & other changes: > http://lists.boost.org/mailman/listinfo.cgi/boost > >

1 0

[Boost MySQL] - Review Period has Ended
by Richard Hodges 21 May '22

21 May '22

Looking at the number of messages in my inbox I see that we have had a great response to the request for reviewers while I have been in transit. Thank you everyone for taking the time to participate. Thank you also to Rubén, as it is apparent from the number of github commits I have seen that he has been very responsive to review comments. Now that I am back home it will take me a little time to collate the results and present. I expect to have this done by the end of the weekend. -- Richard Hodges hodges.r(a)gmail.com office: +44 2032 898 513 home: +376 861 195 mobile: +376 380 212

1 0

Review of Boost.MySQL
by Christian Mazakas 20 May '22

20 May '22

Hello Boost Mailing List, this is my review of Boost.MySQL, ------------ Introduction ------------ Full disclosure, Rubén initially approached me and asked if a MySQL Asio implementation would be wortwhile and I emphatically said, "Yes!" so in some sense, it's impossible for me to be unbiased here. I strongly vote in favor of accepting this library into Boost, for the following set of reasons: * it actually works * it's incredibly idiomatic Asio code * it enables developers to meet business needs * everything else can be built on top of it ---------- Background ---------- I was one of the earlier users of this library, using it going back probably about a year or so ago. I used it at my last job to store records in a local DB. There's probably still a tractor somewhere in a tomato field in California that's running a version of Boost.MySQL on it. Originally, our goal was to have replication working between our deployed machines and a remote server that stored everything "for real". But unfortunately, that requires a bit of DBA voodoo that I'm not capable of so I settled for implementing my own replication logic for the tables where we had to preserve the data. It should also be known that I guess I fall into the "advanced" category of Asio user though that is somewhat contentious. I can write an iniating function okay and preserve executor/allocator correctness. I've also written my fair share of SQL queries as well, working as a webdev for 5+ years so I'm no stranger to MySQL. --------------------------- Discussed Issues & Concerns --------------------------- I've seen some recurring themes during review. Namely, no connection pool, no direct parse-to-struct, no fail-over/healthcheck support. As far as a connection pool is concerned, this should, in my opinion, be considered out-of-scope for an initial acceptance candidate, especially considering the work that was already put into the candidate we have today. It is nice to have one but it's also an open-ended design space and if one was included in the implementation, we'd all be bikeshedding it to death. Instead, I think it's better to let users write what works for them and then gradually we see what a community-driven approach looks like and add that to Boost officially at a later time. It's also worth noting that if you're running against a localhost DB, creating a unix socket connection on something like `/var/run/mysqld/mysqld.sock` is essentially free and you won't even need to use TLS. This helps ameliorate some of the need for a connection pool. Now, in terms of a health-check, this is something the library could add to its `connection` class and have it function identically to how Beast has its websocket streams with their automatic ping/pong semantics. I also saw in a previous email the idea of `void connection::ping()`. I think this should be added before acceptance. If we don't add automatic health-checks for a connection, we should at least give users a proper tool for solving that problem themselves. I will say, however, that it's actually common in Asio to handle reconnection and health-checking yourself. For example, I've delivered Asio-driven applications that communicated with serial ports and CANbus (using SocketCAN). These connection types had reconnection and timeout logic that I manually implemented. These things seem daunting at first but become second-nature once you're accustomed to Asio. Even moreso now that Asio's awaitable types implement `operator||`. As for fail-over, I can't comment too directly as it's something I've never had to deal with in my career. On first thought, I don't know if it'd be exactly the most difficult thing to get a connection to realize it's disconnected and then reconnect to another database from a known whitelist. I'm also under the impression that services like AWS' RDS will simply give you a static hostname or IP address to connect against which acts as a proxy and then the fail-over happens in the background. Maybe I misunderstand the intent of what "fail-over" is so I could simply be uninformed with an opinion here. For parsing directly into a user-defined struct, this is always something nice to have but it shouldn't be a requirement as C++ offers no standardized reflection interface so it's unreasonable to expect this library to accomodate that. This was similarly debated at length during the JSON review and we've learned that a library can still be good while not parsing directly to a structure. ----------------- Personal Opinions ----------------- There's a school of thought that only the lowest-level components should be standardized on a first-pass. Higher-level components can always be built on top of solid primitives. I think Boost.MySQL creates this set of primitives and I think it succeeds. >From a connection, you derive prepared statements or queries. From those, you derive a resultset which is an interface for reading what the DB writes back. Rows derive from the resultset and then values derive from the rows. The surface area of the library is delightfully small and well-contained. I think this library is "simple", for better or worse. I empathize with the side that expresses woes about this simplicity. I think perhaps I'm hardened by my own experiences with Asio so the pains of this library are nothing new to me. To this end, the simplicity of the library is a welcome breath of fresh air from my perspective. The for example, is actually small which is incredibly rare these days. There is one other issue I'd still like to discuss and it's the infamous usage of `std::vector` in interfaces. This is not a problem as a convenience interface as 98% of users are going to use `std::vector` anyway. But it does inhibit Allocators which is unsuitable for acceptance as a Boost library. Everywhere `std::vector` appears, there should be an accompanying version that works with a `Vector&` out-param, i.e. my_company::vector<mysql::row> v; co_await resultset.async_read_all(v); I don't think requiring `vector`-like operations on the type is problematic but preventing control over the allocation is. `error_info&` as an out-param is slightly awkward but I do appreciate that it at least enables us to read the messages about why our queries are wrong. This was my only real complaint about the ergonomics of the library. I would've expected to somehow pull this information from the `error_code` so using an `error_info&` out-param is clunky here. I also don't particularly mind that the backing data of a `value` is tied to the lifetime of a `row` but I can see why this is confusing so perhaps a change in naming is required. --------- Summation --------- In summary, the library is great, it's intuitive, it's idiomatic and I think it belongs in Boost aside other great Asio-based libraries like Beast. At some point, Boost will have its own LAMP/MEAN stack which is the dream. I also realize, I never critiqued the implementation and I think that's because parsing/serializing perf isn't a priority for me and simply working correctly and being clean under valgrind are all that matter so long as the library isn't horrendously slow. If anyone has different opinions on my take, I'd love to hear them. - Christian

2 2

Boost MySQL Review Is In Progress
by Richard Hodges 19 May '22

19 May '22

Dear All, We've had a good response for calls to review this much anticipated library, and there is a lot of interest on Reddit: https://www.reddit.com/r/cpp/comments/unik91/acceptance_review_for_boostmys… . In general the public perception of utility libraries like this in Boost seems popular, and Boost's emphasis on quality is often mentioned. If you are able, I would be more than grateful if you could take a few hours over this delightful weekend to offer a review. The more eyes on this project the better, as if it is successful, it will no doubt prompt submission of similar libraries targeting other popular database and messaging systems. The Boost formal review of the MySQL library started on May 10th, 2022 and will conclude on May 19th, 2022 (inclusive) - In fact, I am able to accept submissions up until the (GMT) morning of May 20th as I'll be traveling on May19th. The library is authored by Rubén Pérez Hidalgo (@anarthal in the CppLang slack). Documentation: https://anarthal.github.io/mysql/index.html Source: https://github.com/anarthal/mysql/ The library is built on the bedrock of Boost.Asio and provides both synchronous and asynchronous client connectors for the MySQL database system. Boost.MySQL is written from the ground up, implementing the entire protocol with no external dependencies beyond the Boost library. It is compatible with MariaDB. Connectivity options include TCP, SSL and Unix Sockets. For async interfaces, examples in the documentation demonstrate full compatibility with all Asio completion handler styles, including: Callbacks:- https://anarthal.github.io/mysql/mysql/examples/query_async_callbacks.html Futures :- https://anarthal.github.io/mysql/mysql/examples/query_async_futures.html Boost.Coroutine :- https://anarthal.github.io/mysql/mysql/examples/query_async_coroutines.html C++20 Coroutines :- https://anarthal.github.io/mysql/mysql/examples/query_async_coroutinescpp20… Rubén has also implemented the Asio protocols for deducing default completion token types :- https://anarthal.github.io/mysql/mysql/examples/default_completion_tokens.h… Reviewing a database connector in depth will require setting up an instance of a MySQL database. Fortunately most (all?) Linux distributions carry a MySQL and/or MariaDB package. MySQL community edition is available for download on all platforms here: https://dev.mysql.com/downloads/ Rubén has spent quite some time in order to bring us this library candidate. The development process has no doubt been a journey of discovery into Asio, its concepts and inner workings. I am sure he has become a fount of knowledge along the way. >From a personal perspective, I was very happy to be asked to manage this review. I hope it will be the first of many more reviews of libraries that tackle business connectivity problems without further dependencies beyond Boost, arguably one of the most trusted foundation libraries available. Please provide in your review information you think is valuable to understand your choice to ACCEPT or REJECT including Describe as a Boost library. Please be explicit about your decision (ACCEPT or REJECT). Some other questions you might want to consider answering: - Will the library bring additional out-of-the-box utility to Boost? - What is your evaluation of the implementation? - What is your evaluation of the documentation? - Will the choice of API abstraction model ease the development of software that must talk to a MySQL database? - Are there any immediate improvements that could be made after acceptance, if acceptance should happen? - Did you try to use the library? With which compiler(s)? Did you have any problems? - How much effort did you put into your evaluation? A glance? A quick reading? In-depth study? - Are you knowledgeable about the problem domain? More information about the Boost Formal Review Process can be found at: http://www.boost.org/community/reviews.html The review is open to anyone who is prepared to put in the work of evaluating and reviewing the library. Prior experience in contributing to Boost reviews is not a requirement. Thank you for your efforts in the Boost community. They are very much appreciated. Richard Hodges - review manager of the proposed Boost.MySQL library Rubén is often available on CppLang Slack and of course by email should you require any clarification not covered by the documentation, as am I. -- Richard Hodges hodges.r(a)gmail.com office: +44 2032 898 513 home: +376 861 195 mobile: +376 380 212

5 6

Review of Boost.MySql
by Marcelo Zimbres Silva 19 May '22

19 May '22

Hi, here is my review. 1) ================================ > https://anarthal.github.io/mysql/mysql/ref/boost__mysql__resultset/async_re… > The handler signature for this operation is > void(boost::mysql::error_code, std::vector<boost::mysql::row>). > etc. Having a std::vector in the signature of the completion handler is unusual in ASIO. In general, large objects should be passed as parameters to the initiating function. Additionally, this operation does not support custom allocators. That wouldn't be a big problem to me if I could reuse the same memory for each request but the interface above prevents me from doing so. It is also a pessimized storage as each row will allocate its own chunk of memory. A single block of memory would be better. 2) ================================ > https://anarthal.github.io/mysql/mysql/ref/boost__mysql__connection/async_q… > The handler signature for this operation is > void(boost::mysql::error_code, boost::mysql::resultset<Stream>) Here the completion handler is returning an io object i.e. boost::mysql::resultset, which is also unusual as far as I can see. Wouldn't it make sense for users to specify a different executor for the resultset? This interface is preventing that. I prefer having the completion handler return config parameters with which I can instantiate a boost::mysql::resultset myself in my desired executor. 3) ================================ > https://github.com/anarthal/mysql/blob/13d3615464f41222052d9ad8fa0c86bcddde… I am not convinced the design of the value class is a good idea. The reason is that users know at compile time what types they are expecting from a given query. That means it would be possible for them to specify that before doing the query. So instead of doing this https://github.com/anarthal/mysql/blob/13d3615464f41222052d9ad8fa0c86bcddde… boost::mysql::row row; while (result.async_read_one(row, additional_info, yield[ec])) users could do boost::mysql::row<T1, T2, T3, T3> row; while (result.async_read_one(row, additional_info, yield[ec])) avoiding unnecessary further copies. In principle it would be even possible to parse directly in a custom data structure, for example, when storing json string in the database. 4) ================================ > https://anarthal.github.io/mysql/mysql/examples.html IMO, the examples are too limited and show variation of the same functionality. For example - Text query, async with callbacks - Text query, async with futures - Text query, async with Boost.Coroutine coroutines - Text query, async with C++20 coroutines - Default completion tokens Here you are showing nice Asio features. I am not meaning you should remove them but if I am not mistaken there is no example that shows - How to keep long lasting connections open (specially for ssl). - Timers for resolve, connect, read and write. - Synchronized writes. - Healthy checks, failover etc. I estimate that to address these points users will be required to write further 500 - 1000 LOC on top of your library. Otherwise they will be stuck on underperforming short lived connections. It would be helpful for users to have a starting point to begin with. There some minor things I have also noticed: 5) ================================ > https://github.com/anarthal/mysql/blob/13d3615464f41222052d9ad8fa0c86bcddde… I expect this function to be O(1). 6) ================================ > https://anarthal.github.io/mysql/mysql/ref/boost__mysql__value/get_std_opti… > https://anarthal.github.io/mysql/mysql/ref/boost__mysql__value/get_optional… Sounds also unusual to have two member functions for the different versions of optional. I suggest using the boost version until we transition to C++17. 7) ================================ > https://anarthal.github.io/mysql/mysql/ref/boost__mysql__errc.html errc should be conditions and not error AFAICS. ============= Review End ================ I vote for conditional acceptance. Points 1, 2, 3 and 4 should be addressed before acceptance. My acceptance criteria sums up to 1) Remove functionality that provides low performance. With coroutines it will be easy for users to loop on async_query to create a std::vector of rows. 2) No IO object should be returned as a parameter of the completion handler. 3) Review the value type. Can it be improved? 4) Improve examples. Regards, Marcelo

10 27

Re: [boost] Review of Boost.MySql
by Ruben Perez 19 May '22

19 May '22

On Wed, 18 May 2022 at 11:05, Kostas Savvidis <ksavvidis(a)gmail.com> wrote: > > Hi Ruben, > > Some people have already suggested making the tcp_ssl connection as the default. In that spirit, it seems to me that it would be better to > remove such prefixes as tcp_ssl from tcp_ssl_prepared_statement and tcp_ssl_resultset > and instead add apropriate prefixes to the non-default connections. Actually, since the only other type is unix socket, > one only need to add the prefix plaintext or nonssl for the noncrypted connections. I like how this sounds. I've updated https://github.com/anarthal/mysql/issues/73 to reflect your comments. Note that we may want to add named_pipe_connection for Windows in the future. > Moreover, is there no technical solution that can completely remove the need to deal with these distinct names for resultset and prepared_statement after the appropriate connection is established? They depend on the Stream type because they have I/O operations. E.g. for a prepared_statement, you need the Stream type to implement the prepared_statement::execute function. Internally, these are really just pointers to the connection object (more or less). One possible option is making prepared_statement and resultset not be I/O objects, and move these functions to the connection object. So the following signatures: prepared_statement::execute(const params<ValueIterator>&) prepared_statement::close() resultset::read_one(row&) Would change to these ones: connection::execute_statement(const prepared_statement&, const params<ValueIterator>&) connection::close_statement(const prepared_statement&) connection::read_row(resultset&, row&) What do you think? Regards, Ruben.

2 2

B2 4.9.0
by René Ferdinand Rivera Morell 19 May '22

19 May '22

There is a new release of B2 and there are some important changes.. * The build-build.jam file, and the build-system rule, used to specify where to find the build system Jam file is no longer used for that purpose. There is now an internal method for finding the corresponding build files relative to the b2 executable. This means that the preferred method for using B2 is to install it first. But, the regular bootstrap use in Boost is still supported. * There is now a Windows MSI installer available that supports both per-user and per-machine installs without having to build it yourself. * Installing the current, and future, B2 allows one to use the newest B2 with current Boost release and most older Boost releases. And of course the same goes for non-Boost projects. For other changes see the release notes < https://github.com/bfgroup/b2/releases/tag/4.9.0> -- -- René Ferdinand Rivera Morell -- Don't Assume Anything -- No Supone Nada -- Robot Dreams - http://robot-dreams.net

4 6

Re: [boost] Boost MySQL Review Starts Today
by Ruben Perez 18 May '22

18 May '22

Forwarding this to the Boost ML (you just replied to me). On Mon, 16 May 2022 at 22:29, Alan de Freitas <alandefreitas(a)gmail.com> wrote: >> >> It is possible to write a Postgres connector with a similar approach. >> >> I indeed wrote a POC available here: >> https://github.com/anarthal/postgres-asio >> The protocols are different, though, so if you follow my approach >> of providing primitives easy to use, yet close to the protocol, >> the APIs will be a little different. The Postgres protocol allows >> finer grain control over statement execution, AFAIR. > > > Oh... That's great. > >> What actions do you think we can take here? Adding a section in the intro >> (https://anarthal.github.io/mysql/mysql/intro.html) about "knowing Asio >> is a requirement to use this library"? Do you think an overview on the protocol >> (handshake, query, metadata and row packets) would really be useful >> for the user at that point? > > > Considering the other reviews I've seen, I think people would like something like "Here's how you can use the library (with synchronous examples, so the user knows they don't need Asio to do what other libraries do)" and only then move to something like "This is how you can use it asynchronously (and by the way you need Asio to do that)". I think the Boost.Process documentation does something like that. I will move the examples on prepared statements forward, so all library functionality is explained before going into the async world. I may also refactor it to try to hide the ssl::context and io_context boilerplate. Besides that, I've got the following example suggestions: https://github.com/anarthal/mysql/issues/87 https://github.com/anarthal/mysql/issues/61 https://github.com/anarthal/mysql/issues/65 Any other ideas are welcome. > >> >> I'm as stuck as you here. What would be the best path here? >> For now, I've tried to contact a former user in the Boost ML who works >> at Oracle, to see whether he can provide me some guidance here. >> Any ideas are welcome. > > > I don't know if being better safe than sorry counts as a solution, but maybe being defensive and trying to come up with new names? Besides the legal issues, which I don't understand, there's the advantage of reflecting it also supports MariaDB. Our contact in Oracle has forwarded our query to the legal department, and I'm waiting for the response. I will wait for that before taking any action. > >> >> At the end of the day, I'd say this is similar to HTTP (half-duplex). > > > Yes. That's correct. I guess the main difference is only the use case. > > An HTTP server might spend its time serving multiple clients asynchronously because they're different connections. > But this same server might also be a MySql client and these SQL requests will need to be serialized to write back to the HTTP clients. > So the database becomes a shared resource being used by a number of asynchronous operations, even though HTTP and MySQL are individually similar. > But this is just something to think about. I guess this is a similar pattern to having the HTTP server calling another HTTP API while serving a request. The user will need to have a number of separate database connections to be able to scale. Connection pooling (https://github.com/anarthal/mysql/issues/19) should be important for this pattern. Using async functions for the database enables the server to keep serving HTTP requests (or whatever other processing it needs to do) while waiting for the MySQL server to answer a query. > >> >> Brought up by another user during review, tracked by >> https://github.com/anarthal/mysql/issues/58. I think the best thing >> here is just passing the vectors by lvalue reference. > > > I would think something like asio::buffer as input and size_t as a result, like asio::read functions, would be better because it allows the preferred data structure to be used. I'd say this pattern is effective for asio because it needs the byte elements in the output buffer to be contiguous. We may be able to use a concept like RowOutputIterator to achieve the same thing. I'm also thinking on making these functions return a custom range that doesn't own the rows (as proposed by Vinnie), but I need to study it more before making a decision. > >> >> Not sure if I follow you. What's the interface you propose here? >> The connection object already holds an internal buffer for requests. >> The library currently performs boost::asio::read call per packet, >> but I'm working in optimizing that to use read_some instead. >> With that approach, read_one may get the row from the buffer >> without performing any read at all. Is that what you mean? > > > Yes. If it read whatever it can to an internal buffer first, then read_some probably fixes that. I will have read_one read from the buffer if possible, and I'm thinking of implementing a read_some function for rows, but as I mentioned before, I'm still designing. > >> >> If we go down the compile-time parsing strategy, together with the buffering >> I described above, making it owning may be the best option. > > > Mmm... I hadn't considered that. That's correct. > Although I do not dislike the idea of it being a "view", because it's the most common use case anyway. > The only thing I didn't like was a "view" called "value". I've been given points in both directions. Would naming it sql_value_view or field_view make things better? Updated https://github.com/anarthal/mysql/issues/85. > >> >> First of all, I'd like to know your thoughts about the conversion >> functionality in value >> (https://anarthal.github.io/mysql/mysql/values.html#mysql.values.conversions) >> Do you think it's helpful for the user? > > > Yes. This looks good. > >> >> If it wasn't because of conversions, >> we could replace value::get_optional and value::get_std_optional for >> something similar to variant2::get_if, returning a pointer to the >> underlying value, >> avoiding the problem altogether. This could make even more sense if we make >> the value owning, as value::get_optional<std::string> would end up copying >> the underlying string, which doesn't make a lot of sense. > > > Yes. It depends on your next decisions. In any case, it would remove get_std_optional if get_optional is not removed. > >> Calling value::is_convertible_to, then value::get will perform the conversion >> twice, while value::get_optional will do it just once. It isn't >> horribly inefficient, >> though - maybe this section is causing more confusion than aid. > > > I'm not sure I'm following. So calling value::is_convertible_to actually needs to perform the conversion and then throw it away? It does. However, the conversions should be really fast - some int comparisons in the worst case. No memory allocations or expensive calls. > >> >> 4) Make value::get return a reference instead of a value. > > > You might have some problems keeping the implicit conversions here. I don't know what other libraries do. That would only be possible if we removed conversions. > >> I'm thinking on calling it query_operation or query_result. > > > I'm terrible at coming up with names. But query_operation or query_result look good. > >> >> > - The names "tcp_ssl_prepared_statement", "tcp_ssl_<component>" are quite >> > repetitive and verbose in the documentation. Maybe "tcp_ssl" could become a >> > namespace? Isn't that what Asio does for the ssl related objects? Some >> > names might be reduced if there's no alternative. For instance, >> > "prepared_statement" might become "statement" since there's no >> > "unprepared_statement". >> >> Would having type aliases under connection (i.e. connection::resultset_type) >> alleviate your concerns? > > > I think I would try to replicate whatever pattern Asio has. But it's really up to you. I've updated https://github.com/anarthal/mysql/issues/73 to add your suggestions. > >> >> Definitely. https://github.com/anarthal/mysql/issues/74 tracks it. >> How would variadic operator() work with async functions? > > > Yes. I'm not sure it has to be operator(). An explicit name, like execute, could be better. execute might become ambiguous though. To follow Asio's convention, I'd keep it a function like execute(). Should be possible to make it work, AFAIK. > >> >> Not aware there was a standard format. Could you please point >> me out to the relevant documentation? > > > I think this is the common template: https://github.com/boostorg/boost-ci > IRRC, some libraries, such as beast, have two paths: One path with find_package(Boost) when not in boost/libs and one path that considers it to be in boost/libs. > The template only has the second path. I haven't been able to locate the template - I can see a couple CMakeLists.txt but they seem test-related. Could you please point me to the specific file? > >> They are from the perspective that they require a real server to run. > > > Yes. They require lots of things but testing integration it's not their purpose or something specific to them. If their purpose was to test integration (with what?), then the examples would be wasteful because you don't need them to test integration. For instance, https://github.com/boostorg/boost-ci/tree/master/test/cmake_test includes a Cmake integration test, with only what we need to test CMake integration. (This is a minor point anyway) It could be worth changing the code to only *run* the examples if BOOST_MYSQL_INTEGRATION_TESTS is defined (as a signal of "hey, the user has a correctly configured server and wants to run all the tests she can"), but always build them, regardless of BOOST_MYSQL_INTEGRATION_TESTS. > >> >> > - The documentation refers to topics that haven't been explained yet. Maybe >> > "value" could be explained after "row", and "row" could be explained after >> > "resultset" and "resultset" after "queries". >> >> The basic concepts have been covered in the tutorial, which goes first. >> So at this point the user should know the *basic* concept of what a value, >> a row and a resultset is. The rest of the sections go deeper in these concepts. >> Did you find section order counter-intuitive, even reading the tutorial first? > > > I don't find it counter-intuitive. I think the mental model of (roughly) values < row < results < statements < DB is clear. > > The point here is probably that the snippets mysql/values.html represent stuff the user probably doesn't need and the following sections don't have many snippets. > At the same time, for instance, you referred to the tutorial when you thought I didn't understand the documentation. > > So I have the impression that the model of "values -> row -> results -> statements -> DB" makes it difficult to provide useful examples, while "DB -> statements -> results -> row -> values" makes the documentation more pleasant to read. > This is probably more important as the documentation grows with time. Technically, both options are fine. I see your point. Raised https://github.com/anarthal/mysql/issues/90. > >> >> No. In Asio, you always get two overloads for sync operations. > > > Oh... of course. I forgot the subsection was only talking about synchronous functions when I was reading that sentence. > >> >> I'd say this is personal preference. I will explore the tabular format >> when I have the chance. > > > The text format is not bad if you want to find the C++ type for a SQL type. > If you're looking for something with a different "column" as a reference, then you have to scan a lot of text to find what you want. Ah, do you mean adding a table that summarizes the mappings, additionally to what is currently written? That could be useful. Raised https://github.com/anarthal/mysql/issues/91. > >> >> Could you please be a little more explicit on your second point? >> Would adding a struct/tuple interface >> (https://github.com/anarthal/mysql/issues/60) > > > The tuple interface is something (really) nice to have but I don't think this should be a requirement for acceptance. > > On the other hand, the possibility of reusing memory (i.e. not depending on vector on the callbacks) seems urgent at this point. > This is what asio already does for its functions with asio::buffer so you don't depend on a specific data structure, like vector. > I think other people have mentioned the same issue in their reviews, and it seems problematic to bake this into the API at this point. > > For instance, if we replicate the asio interface, we would have something *similar* to: > > std::vector<boost::mysql::row> employees; > std::size_t n = result.read_all(employees); > > I'm not sure this example has the *best* API possible, but Asio functions tend to have this format. For instance, consider Boost.Beast: > > std::size_t n = http::read(stream, buffer, res); > > https://www.boost.org/doc/libs/1_79_0/libs/beast/doc/html/beast/ref/boost__… > > Same format. The buffers are input parameters and the function returns size_t. And the callback gives you error_code and size_t. > > Another small advantage of this format is that the callback function will match those of asio::read/asio::write, which is useful with asio::coroutine. > https://www.boost.org/doc/libs/1_79_0/doc/html/boost_asio/reference/corouti… Got it. I have to decide which API it will get, but it will allow for memory re-use. > >> >> and making it owning (https://github.com/anarthal/mysql/issues/85) > > > I don't even think making it own the data is important. I understand the value of making it a view. But that depends on other reviews. > I just care that something is done about it. Either making it owning or changing the name. Got it too. > >> >> Kind regards, > > > Thanks again for the library. Great work. > >> >> Ruben. > > > > -- > Alan Freitas > https://github.com/alandefreitas Thank you for your comments.

1 0