- Boost - lists.boost.org

Ruben's Hash2 review
by Ruben Perez 13 Dec '24

13 Dec '24

Hi all, This is my review of the proposed Boost.Hash2. Thanks Peter and Chris for submitting the library, and Matt for managing the review. - What is your evaluation of the design? Very good. This is how hashing C++ objects should work, as it properly separates concerns between types and hash functions. I think usability could be improved if the template class hash shown as an example was included as part of the library and introduced much earlier in the documentation. In general, the HashAlgorithm concept makes sense. I'm not convinced about the HashAlgorithm::result() interface, for two reasons: * It makes integration of third-party hashes more difficult, as already noted by other reviewers. * The naming suggests an accessor, rather than a modifier. I'd consider splitting it into two functions, maybe finalize_block() and result(). But I can also see the benefits of the approach taken. I miss some higher-level features when working with untyped byte sequences. While this doesn't seem the main value proposition of the library, it's listed as a valid use case. See below for more info. These are nice-to-haves rather than requirements, though. I was initially surprised to find out that the library was header-only, as I didn't know that compile-time hashing was something worth it. I was initially worried about compile-times. I built one of the examples with compile-time profiling (-ftime-trace) and the library is pretty lightweight, so I have no problems with the header-only design. Have you considered somehow supporting hashing for objects describable via Boost.Pfr? I had it requested by some users in MySQL even when I had built-in Boost.Describe support, so it may get requested here, too. - What is your evaluation of the implementation? The parts where I've peaked in are good. Some comments would have helped me, but in general I've been able to follow. The test suite looks pretty comprehensive, too. Some fuzz tests for the cryptographic algorithms would be appreciated (I can help if required). As other reviewers have commented, the algorithm for hashing unordered ranges can use some improvements to make it more secure. I'm no security expert and can't judge how important this is, but I'm pretty sure the author will solve this promptly. The implementation of cryptographic functions is currently slower than solutions like OpenSSL [4], but the author has manifested the possibility of closing this gap. It would be nice having HashAlgorithm modelled as a C++20 concept, although the required macro machinery required to perform the checking only if concepts are supported (like BOOST_ASIO_COMPLETION_TOKEN_FOR does) may be too much effort for the benefit it brings. Regarding Boost integration, a build.jam should likely be added, so the library integrates in the new modular Boost infrastructure (although I've been able to use the library fine without it). This can be done after acceptance. As a matter of curiosity: which compilers do not support __builtin_is_constant_evaluated? (as they will always get the slow memcpy implementation, even at runtime). - What is your evaluation of the documentation? The documentation is good, and explains rationales pretty well, but the order could be improved: * Most users will likely want to use the existing algorithms with their unordered containers, so this should be explained first. * The next block of users will likely want to use the existing cryptographic hash algorithms to untyped byte sequences, so this should be next. * The rest can be left as is. I miss the motivation for the flavor construct in the docs. If I understood correctly, it's only relevant if you plan to somehow serialize your hashes. Maybe an example could help. The library is also suitable for use cases that combine the C++ object hashing and untyped byte sequence hashing parts, as explained by the author here: https://lists.boost.org/Archives/boost/2024/12/258765.php. Such cases are the rationale behind having cryptographic functions within the same umbrella as non-cryptographic ones. I'd suggest adding these to the docs. In general, I dislike single-page docs (which unfortunately seem pretty popular in Boost), as they're difficult to search. I also miss linking (e.g. hash_append should contain a link to its description in the reference). It would also be beneficial to explain the library's relationship with boost::hash / Boost.ContainerHash. In general, making it more approachable for non-expert users would be good. - What is your evaluation of the potential usefulness of the library? Do you already use it in industry? Robust, extensible C++ object hashing is very useful. My immediate use case is related to hashing untyped byte sequences, though. MySQL/MariaDB authentication involve a challenge-response protocol that involves hashing the user-supplied password with a server-supplied challenge, using either SHA1 (sigh) or SHA256 [1]. I'm currently using OpenSSL, as I use it for TLS, too. This library would allow me to make the OpenSSL dependency optional (which I've been requested). I'm also planning on writing a Postgres protocol library. It will likely need to support SCRAM-SHA256 authentication, similar to what I've described. This library could be handy for this. - Did you try to use the library? With which compiler(s)? Did you have any problems? I've re-written the password hashing functionality in Boost.MySQL to use the proposed library [2]. Everything has been very smooth. The test suite passes for all the compilers I support [3], which is really good. I'd have liked some utilities on top of HashAlgorithm to make hashing byte sequences safer and easier. These utilities would *not* imply any change in the HashAlgorithm interface: * Adding a contiguous range of bytes to an algorithm currently supports only a pointer + size pair. This is the right choice for HashAlgorithm, but having a helper function on top of it would be great, as proposed by other reviewers. This would make my code go from: // password is a string_view hash2::sha1_160 hasher1; hasher1.update(password.data(), password.size()); auto password_sha1 = hasher1.result(); To: hash2::sha1_160 hasher1; hash2::hash_add_bytes(hasher1, password); auto password_sha1 = hasher1.result(); * Adding a function that computes the hash of an entire byte sequence, as a range, could be useful. OpenSSL has this as SHA1() and SHA256(). This would make my code go from: hash2::sha1_160 hasher1; hasher1.update(password.data(), password.size()); auto password_sha1 = hasher1.result(); To: auto password_sha1 = hash2::compute_hash(hash2::sha1_160(), password); These are just nice to have and trivially implemented in user-land, if required. The code is already good as is today. - How much effort did you put into your evaluation? A glance? A quick reading? In-depth study? I have dedicated around 8 hours to study the documentation, build the library and examples, integrate it into Boost.MySQL, update my CIs to run my tests with the library, and write this review. - Are you knowledgeable about the problem domain? I am a mere user of hash algorithms, as I usually write networking code. I don't know about their internals. I haven't authored associative containers, either. - Affiliation disclosure I am currently affiliated with the C++ Alliance. - Final decision My recommendation is to CONDITIONALLY ACCEPT the library into Boost. My only acceptance condition is adding the hash template class developed in the examples to the library. The benefit of having the library in Boost is big and overcomes any concerns I might have. Many thanks to both Peter and Matt. Kind regards, Ruben. [1] https://dev.mysql.com/doc/dev/mysql-server/8.4.3/page_caching_sha2_authenti… [2] https://github.com/boostorg/mysql/pull/389 [3] https://github.com/boostorg/mysql/blob/c438f26731e36c2db6457705ec5dbb9f7657… [4] https://lists.boost.org/Archives/boost/2024/12/258687.php

4 4

Re: [boost] [hash2] My Review
by Vinnie Falco 12 Dec '24

12 Dec '24

On Sat, Dec 7, 2024 at 5:47 AM Matt Borland via Boost <boost(a)lists.boost.org> wrote: > The review of Hash2 by Peter Dimov and Christian Mazakas begins today > Saturday, December 7th through Sunday December 15th, 2024. > > Code: https://github.com/pdimov/hash2 > Docs: https://pdimov.github.io/hash2/ Disclaimer: I am affiliated with The C++ Alliance and I am also a co-author of N3980 (the paper referenced by the library). This is my review of Hash2. Spoiler alert, my suggestion is to: ACCEPT While most of my review is dedicated to discussing some concerns I have about some aspects of the library, it is clear that Boost as a collection is better with Hash2 than without it. Also, Peter is going to do what Peter is going to do so why fight it. The uncontroversial parts are stated first, and after lies my assessment of some questionable design choices. > - What is your evaluation of the implementation? > I looked over some of the implementation to better understand how certain things work. The tests pass on CI and the test code looks comprehensive. Peter is not in the habit of producing anything less than high-quality implementations and this library is no exception. I don't know which parts are from Peter and which are from Christian and the quality appears high throughout. > - What is your evaluation of the documentation? > I think the documentation could be better in places. The named requirements exposition is too informal for my taste and talks about things that are not strictly relevant, such as zero-padding. The library alludes to cryptographic usage yet glosses over some details which surfaced during mailing list discussion (more on that below). It would be nice if there was some exposition up front explaining which groups of users the library is for, and how each group uses the library. > - What is your evaluation of the potential usefulness > of the library? Do you already use it in industry? > Not only is the library useful, but the C++ Standard is poorer for not having a modern hash solution. Competing approaches were developed by Google and N3980, then promptly abandoned. > - Did you try to use the library? With which compiler(s)? Did > you have any problems? > I tried to build a Visual Studio Solution from the CMakeLists.txt after cloning hash2 into my superproject, with this command line: cmake -G "Visual Studio 17 2022" -A x64 -B bin64 -DCMAKE_TOOLCHAIN_FILE="C:/vcpkg/scripts/buildsystems/vcpkg.cmake" -DVCPKG_CHAINLOAD_TOOLCHAIN_FILE="C:/users/vinnie/src/toolchains/msvc.cmake" It started downloading things from the internet (FetchContent I guess) and then failed. I was not able to generate a solution. I asked Peter about this and he said it was not supported. It would be nice if it was. > - How much effort did you put into your evaluation? A glance? A quick reading? In-depth study? > I focused primarily on the ways in which the library differs from the original paper N3980. I looked at the relevant documentation, interfaces, and implementation, and engaged the author on the mailing list. - Are you knowledgeable about the problem domain? Fairly knowledgeable. - What is your evaluation of the design? This is where I have concerns. The audience of users for Hash2 can be divided into roughly these groups: 1. Authors of types who just want to make objects of their type hashable without having to learn about hash algorithms cryptography, and users who want higher quality results when working with unordered containers. Hash2 is basically perfect for these two groups of users (which often overlap) although the documentation could be improved to speak more directly to them. 2. Users who wish to calculate a cryptographic digest for objects of Hash2-enabled types, or for untyped binary data. Hash2 is also almost perfect for these use cases, and as before the documentation could be more explicit about these cases. 3. Users who wish to adapt existing code to meet the HashAlgorithm requirements from Hash2. This is where I believe the library has some design problems. An unfortunate consequence of being right almost all the time, is that it becomes more unlikely to recognize when you are wrong. I will assume that the reader is at least somewhat familiar with N3980 and some of the arguments I made about the HashAlgorithm named requirements on the mailing list. Hash2 attempts to improve the quality of life for users by mandating additional requirements for the HashAlgorithm concept which may be difficult or impossible to meet for some existing libraries. For example, it requires that every HashAlgorithm support: * Seeded construction from std::uint64_t * Keyed construction from binary data * Copy construction * Digest extension from repeated calls to result() * Interleaving of calls to update() and result() * ...Future stuff that the author alludes to These issues above are already discussed extensively on the mailing list and I can't be bothered to spend hours putting it all together into a beautiful post the way that some of the other reviewers have done (Andrey Semashev and Samuel Neves come to mind). The burdensome additional named requirements of Hash2's HashAlgorithm ensure that no casual users will be authoring them. Or worse, they will get them wrong and the results will not have the same guarantees as the underlying algorithms. Right now there are approximately 1,649,127 implementations of cryptographic and non-cryptographic hash algorithms. I think it might be more prudent for Hash2 to offer a system of traits for identifying which algorithms are capable of what, such as CopyConstruction, digest extension, and so on, and give the user tools which only work in safe ways given the traits of a hash algorithm. This would make it easier to write HashAlgorithms. And as a consequence we might get many more of them rather than fewer. Realistically I don't expect the library to change, and a proliferation of user-created HashAlgorithms is probably conditional on the Boost library collection being much more popular than it currently is. This is partly why I have decided to just accept the library. And I could be wrong, and Peter's unwavering insistence that users get HashAlgorithms which all offer the same quality of service might turn out to be better. Time will tell. Thanks

2 1

[hash2] Difference from paper: HashAlgorithm interface vs. function object
by Andrey Semashev 12 Dec '24

12 Dec '24

Hi, The HashAlgorithm interface defined in Boost.Hash2 is notably different from the one described in N3980 (https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2014/n3980.html) Specifically, the paper defines HashAlgorithm as a function object, whose operator() is equivalent to Boost.Hash2's HashAlgorithm::update method. The function object also provides a conversion operator to result_type, which is equivalent to Boost.Hash2's HashAlgorithm::result method. Is this discrepancy intentional and are there plans (on the paper authors' or the library authors' side) to eliminate it? Personally, I don't think a function object is the right interface for a hash algorithm. Update and finalize are two distinct operations provided by hash algorithms and both should be performed explicitly by the user. I don't think a conversion operator is the right name for finalize. Conversion operators are normally used when one type can meaningfully "impersonate" another, and this is not the case here. The function object is also misusing result_type, as its operator() doesn't return it. The paper discusses that the fact that a hash algorithm is a function object can be used to wrap it in std::function, which can be useful with pimpl idiom and for the purpose of type erasure. I don't think this will actually work, at least not as simple as the paper describes it, because std::function won't allow to extract the computed hash digest (i.e. you won't be able to call the type conversion operator on the wrapped function object). I think, the user will have to write some wrapper classes anyway to retain access to the wrapped hash algorithm. However, this is still an interesting use case. Does Boost.Hash2 intend to provide utilities to address it? For example, Boost.Hash2 could provide a hash-algorithm-agnostic interface for feeding an algorithm with input data, and a wrapper for an external algorithm object that would implement this interface. Something along these lines: class polymorphic_hash_algorithm { public: polymorphic_hash_algorithm( polymorphic_hash_algorithm const&) = delete; polymorphic_hash_algorithm& operator=( polymorphic_hash_algorithm const&) = delete; virtual void update(const void* data, size_t size) = 0; protected: polymorphic_hash_algorithm() = default; ~polymorphic_hash_algorithm() = default; }; template< typename Hash > class polymorphic_hash_algorithm_wrapper : public polymorphic_hash_algorithm { public: using hash_type = Hash; using result_type = typename hash_type::result_type; static constexpr size_t block_size = hash_type::block_size; private: Hash& h; public: explicit polymorphic_hash_algorithm_wrapper(hash_type& h) noexcept : h(h) {} void update(const void* data, size_t size) override { h.update(data, size); } result_type result() { return h.result(); } }; This way, polymorphic_hash_algorithm could be used to abstract away the hash algorithm type while still be compatible with hash_append and friends. As a side note, this would be another use case where the hash algorithm passed to hash_append is non-copyable. And while copyability could be added, most likely it wouldn't be free.

6 11

The Master branch is now open for changes
by Marshall Clow 12 Dec '24

12 Dec '24

Thanks to everyone for their patience. The next release cycle will begin in late February. — The release managers.

1 0

[hash2] Question about cryptographic guarantees of hash_append_unordered_range
by Ivan Matek 12 Dec '24

12 Dec '24

Hi everyone, Implementation uses uint64_t in which it sums into hashes of particular elements(each computed with "fresh" hasher so ordering does not matter), then it finally hashes that sum and range size into original passed hasher. I see no problem with logic this since it cleverly exploits the fact that uint64_t overflow is defined and that sum is commutative. But I wonder if this preserves the cryptographic quality of original hash. For example if we use sha2 it seems to me that finding collisions becomes easier since sum is "only" 64 bits, and length is "only" 64 bits(or 32 on some systems), and may be even less if attacker knows unordered range was not super long, e.g. number of items was less than 10000. Does somebody know more about this particular domain and could point me to some resource to learn more? I know it was in the original proposal, but it is possible original proposal was not focusing only on cryptographically secure algorithms. I was unable to google any guarantees, maybe because idk what keywords to use. :)

9 17

[release] Boost 1.87.0
by Marshall Clow 12 Dec '24

12 Dec '24

Release 1.87.0 of the Boost C++ Libraries is now available. These open-source libraries work well with the C++ Standard Library, and are usable across a broad spectrum of applications. The Boost license encourages both commercial and non-commercial use. This release contains two new libraries: * Boost.Parser: A framework for building parsers, from Zach Laine. The release also contains numerous enhancements and bug fixes for existing libraries. For details, including download links, see <http://www.boost.org/users/news/version_1.87.0> You can also download directly from: <https://boostorg.jfrog.io/artifactory/main/release/1.87.0/> To install this release on your system, see <http://www.boost.org/doc/libs/release/more/getting_started/index.html> Thanks to everyone who participated in this release. -- The Boost Release Team Marshall Clow, Glen Fernandes, Ion Gaztañaga

2 2

[Hash2] Current list of items brought up during review
by Peter Dimov 11 Dec '24

11 Dec '24

This is what I've scribbled so far as a list of to-do items: - add hash_append_range taking a range - add a "provider" argument (providing the hash_append API) to tag_invoke - example of hashing boost::json::value - maybe change argument of tag_invoke to take the value by pointer, rather than by reference - maybe expose has_tag_invoke to users so they can specialize - make byte sequence seed constructor take `void const*` as well - change get_integral_result to take a Hash, instead of h.result(), so that it can invoke result() more than once - a user-accessible way to test whether a hash algorithm conforms to the requirements - add a variadic hash_append overload - hash_append_unordered_range: use more than 64 bits for intermediate hashes (uint64_t[4]); improve somehow https://lists.boost.org/Archives/boost/2024/12/258643.php - acknowledge https://github.com/HowardHinnant/hash_append - in documentation, focus more on the std::hash case, rather than on the md5sum case - in FNV-1a byte seed constructor, include the seed size - change block_size to std::size_t

2 1

Re: [boost] [Hash2] Result extension
by Peter Dimov 11 Dec '24

11 Dec '24

Vinnie Falco wrote: > Sigh.. copy/paste typo. > > /** Return a seeded HashAlgorithm > */ > template< class HashAlgorithm > > HashAlgorithm make_seeded( > unsigned char const* seed, std::size_t n ) > { > if constexpr(std::is_constructible<HashAlgorithm, > unsigned char const*, std::size_t>) > return HashAlgorithm(seed, n); > else > { > HashAlgorithm h; > hash_append(h, seed, n); You can in principle do something like that (except fix it to compile) but (much) better practices would be - hash.update( seed, n ) - encode `n` as 64 bit little endian in 8 bytes - hash.update these bytes - hash.update( "\x80", 1 ) - hash.update( "\x00", 1 ) - hash.update with as many zeroes as needed to reach a multiple of HashAlgorithm::block_size That's, incidentally, exactly what `update(p, n); result();` does. (What an amazing coincidence.) Note that reaching a multiple of block_size is important, because it ensures that the secret key you passed as the seed is not left in the internal buffer of the hash algorithm, visible in memory dumps. The test https://github.com/pdimov/hash2/blob/develop/test/plaintext_leak.cpp tries to catch these mistakes. That's why this line https://github.com/pdimov/hash2/blob/7a25f8518692b657e9272884519519fbaca2ec… is needed, for example. > return h; > } > } > > Thanks >

4 6

Github Actions CI updates - e.g. Node 20
by Alexander Grund 10 Dec '24

10 Dec '24

Dear maintainers, I made a bunch of PRs to fix the Github Actions CI for almost 40 repositories. Opening a PR in GitHub is manual work while I was able to apply similar changes over all those repositories using a script which commits to a single branch (and hence PR) per repo. Hence the PRs contain a few related changes that in combination allow the CI to pass again. To avoid repeating this in each PR I wanted to shortly explain the changes here: 1. macos-12 runners are dropped by github and jobs won't start anymore. at least macos-13 needs to be used 2. Ubuntu 23.* doesn't work anymore as e.g. some APT repositories don't exist anymore. Ubuntu 24.04 can be used instead 3. clang 17 (and partly 16) don't work in C++23/2b mode with the stdlibc++ (12) installed on ubuntu 24.04, clang 18+ and/or the other C++ standards can still be used 4. Support for Node 16 was recently fully dropped by Github Actions so the @v3 actions must be upgraded to @v4, which require Node 20. The workaround using $ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION has no effect anymore 5. The node 20 version available by default conflicts with the glibc version in the Ubuntu 16/18 container images used for testing older compilers. I addressed all of them with last one being the major one: Installing a compatible version inside a custom image to be used instead doesn't work because GHA mounts the node 20 installation from the host os. The solution I implemented downloads an unofficial node 20 built against the correct/older glibc and creates 2 mount points such that we can put that node onto the host runner and mount that instead of what github gives us using the idea from https://github.com/dixyes/ghactionsplay/blob/bf889ee2fed8dcd72cc8e32bfd4789… To avoid overloading the unofficial-builds.nodejs.org server we use a copy hosted at boost.io Download and extraction requires curl and xz-utils to be installed which I added where required. Additionally I noticed additional failures due to use of C++03 where the dependencies require C++11, or of C++0x using GCC < 4.9 where e.g. C++11 `alignof`, required by the Boost.SmartPtr dependency, is not available. For (only) those I removed the C++03 and/or GCC 4.7/4.8 jobs (which cannot succeed). I think combining those changes into a single PR avoids a lot of additional manual work by me and allows to verify that the changes are sufficient to make CI pass again. Keeping e.g. C++03 builds, that cannot succeed due to dependencies, makes verifying the success of the Node 20 fix harder/less clear If anyone prefers to separate those changes into multiple PRs please mention that in the relevant PR for me to keep track of, or cherry-pick the individual changes. However it might be sufficient to check the status of the CI for the individual commits to compare the state (success/failure) for the jobs fixed or removed by the next commit. I realize maintaining the CI is a PITA and hope this helps. Regards, Alex

6 8

[Release] Boost 1.87.7 Release Candidate 1 is available
by Marshall Clow 09 Dec '24

09 Dec '24

The first release candidates for the 1.87.0 release are now available at: <https://boostorg.jfrog.io/artifactory/main/release/1.87.0/source/> The SHA256 checksums are as follows: f55c340aa49763b1925ccf02b2e83f35fdcf634c9d5164a2acb87540173c741d boost_1_87_0_rc1.tar.gz af57be25cb4c4f4b413ed692fe378affb4352ea50fbe294a11ef548f4d527d89 boost_1_87_0_rc1.tar.bz2 f367a218d27b23b568956bacf56d08d2c1d9647211337136be8cac5cb2ff0828 boost_1_87_0_rc1.zip db6b7d77a2286ebe79d4aa7252c08b051d271fd8b013b6b18bc7a966552c1139 boost_1_87_0_rc1.7z As always, the release managers would appreciate it if you download the candidate of your choice and give building it a try. Please report both success and failure, and anything else that is noteworthy. -- The Release managers

8 11