December 2024 - Boost - lists.boost.org

[Hash2] Current list of items brought up during review
by Peter Dimov 11 Dec '24

11 Dec '24

This is what I've scribbled so far as a list of to-do items: - add hash_append_range taking a range - add a "provider" argument (providing the hash_append API) to tag_invoke - example of hashing boost::json::value - maybe change argument of tag_invoke to take the value by pointer, rather than by reference - maybe expose has_tag_invoke to users so they can specialize - make byte sequence seed constructor take `void const*` as well - change get_integral_result to take a Hash, instead of h.result(), so that it can invoke result() more than once - a user-accessible way to test whether a hash algorithm conforms to the requirements - add a variadic hash_append overload - hash_append_unordered_range: use more than 64 bits for intermediate hashes (uint64_t[4]); improve somehow https://lists.boost.org/Archives/boost/2024/12/258643.php - acknowledge https://github.com/HowardHinnant/hash_append - in documentation, focus more on the std::hash case, rather than on the md5sum case - in FNV-1a byte seed constructor, include the seed size - change block_size to std::size_t

2 1

Re: [boost] [Hash2] Result extension
by Peter Dimov 11 Dec '24

11 Dec '24

Vinnie Falco wrote: > Sigh.. copy/paste typo. > > /** Return a seeded HashAlgorithm > */ > template< class HashAlgorithm > > HashAlgorithm make_seeded( > unsigned char const* seed, std::size_t n ) > { > if constexpr(std::is_constructible<HashAlgorithm, > unsigned char const*, std::size_t>) > return HashAlgorithm(seed, n); > else > { > HashAlgorithm h; > hash_append(h, seed, n); You can in principle do something like that (except fix it to compile) but (much) better practices would be - hash.update( seed, n ) - encode `n` as 64 bit little endian in 8 bytes - hash.update these bytes - hash.update( "\x80", 1 ) - hash.update( "\x00", 1 ) - hash.update with as many zeroes as needed to reach a multiple of HashAlgorithm::block_size That's, incidentally, exactly what `update(p, n); result();` does. (What an amazing coincidence.) Note that reaching a multiple of block_size is important, because it ensures that the secret key you passed as the seed is not left in the internal buffer of the hash algorithm, visible in memory dumps. The test https://github.com/pdimov/hash2/blob/develop/test/plaintext_leak.cpp tries to catch these mistakes. That's why this line https://github.com/pdimov/hash2/blob/7a25f8518692b657e9272884519519fbaca2ec… is needed, for example. > return h; > } > } > > Thanks >

4 6

Github Actions CI updates - e.g. Node 20
by Alexander Grund 10 Dec '24

10 Dec '24

Dear maintainers, I made a bunch of PRs to fix the Github Actions CI for almost 40 repositories. Opening a PR in GitHub is manual work while I was able to apply similar changes over all those repositories using a script which commits to a single branch (and hence PR) per repo. Hence the PRs contain a few related changes that in combination allow the CI to pass again. To avoid repeating this in each PR I wanted to shortly explain the changes here: 1. macos-12 runners are dropped by github and jobs won't start anymore. at least macos-13 needs to be used 2. Ubuntu 23.* doesn't work anymore as e.g. some APT repositories don't exist anymore. Ubuntu 24.04 can be used instead 3. clang 17 (and partly 16) don't work in C++23/2b mode with the stdlibc++ (12) installed on ubuntu 24.04, clang 18+ and/or the other C++ standards can still be used 4. Support for Node 16 was recently fully dropped by Github Actions so the @v3 actions must be upgraded to @v4, which require Node 20. The workaround using $ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION has no effect anymore 5. The node 20 version available by default conflicts with the glibc version in the Ubuntu 16/18 container images used for testing older compilers. I addressed all of them with last one being the major one: Installing a compatible version inside a custom image to be used instead doesn't work because GHA mounts the node 20 installation from the host os. The solution I implemented downloads an unofficial node 20 built against the correct/older glibc and creates 2 mount points such that we can put that node onto the host runner and mount that instead of what github gives us using the idea from https://github.com/dixyes/ghactionsplay/blob/bf889ee2fed8dcd72cc8e32bfd4789… To avoid overloading the unofficial-builds.nodejs.org server we use a copy hosted at boost.io Download and extraction requires curl and xz-utils to be installed which I added where required. Additionally I noticed additional failures due to use of C++03 where the dependencies require C++11, or of C++0x using GCC < 4.9 where e.g. C++11 `alignof`, required by the Boost.SmartPtr dependency, is not available. For (only) those I removed the C++03 and/or GCC 4.7/4.8 jobs (which cannot succeed). I think combining those changes into a single PR avoids a lot of additional manual work by me and allows to verify that the changes are sufficient to make CI pass again. Keeping e.g. C++03 builds, that cannot succeed due to dependencies, makes verifying the success of the Node 20 fix harder/less clear If anyone prefers to separate those changes into multiple PRs please mention that in the relevant PR for me to keep track of, or cherry-pick the individual changes. However it might be sufficient to check the status of the CI for the individual commits to compare the state (success/failure) for the jobs fixed or removed by the next commit. I realize maintaining the CI is a PITA and hope this helps. Regards, Alex

6 8

[Release] Boost 1.87.7 Release Candidate 1 is available
by Marshall Clow 09 Dec '24

09 Dec '24

The first release candidates for the 1.87.0 release are now available at: <https://boostorg.jfrog.io/artifactory/main/release/1.87.0/source/> The SHA256 checksums are as follows: f55c340aa49763b1925ccf02b2e83f35fdcf634c9d5164a2acb87540173c741d boost_1_87_0_rc1.tar.gz af57be25cb4c4f4b413ed692fe378affb4352ea50fbe294a11ef548f4d527d89 boost_1_87_0_rc1.tar.bz2 f367a218d27b23b568956bacf56d08d2c1d9647211337136be8cac5cb2ff0828 boost_1_87_0_rc1.zip db6b7d77a2286ebe79d4aa7252c08b051d271fd8b013b6b18bc7a966552c1139 boost_1_87_0_rc1.7z As always, the release managers would appreciate it if you download the candidate of your choice and give building it a try. Please report both success and failure, and anything else that is noteworthy. -- The Release managers

8 11

[Hash2] Result extension
by Peter Dimov 09 Dec '24

09 Dec '24

Vinnie Falco asked me the following on Slack: > I would ask, what is the motivating use-case for calling result > twice? This is not explained in the docs and no examples are > given. In fact, the one example given says "not to do this" Calling result() twice (or more times) provides result extension; the ability to extract variable number of bits from a hash algorithm, instead of a fixed size value (e.g. 64 bit.) This is in fact stated in the docs here https://pdimov.github.io/hash2/doc/html/hash2.html#hashing_bytes_result > Note that result is non-const, because it changes the internal > state. It’s allowed for result to be called more than once; > subsequent calls perform the state finalization again and as a > result produce a pseudorandom sequence of result_type values. > This can be used to effectively extend the output of the hash > function. For example, a 256 bit result can be obtained from a > hash algorithm whose result_type is 64 bit, by calling result four > times. and there is an example of doing that here https://pdimov.github.io/hash2/doc/html/hash2.html#example_result_extension All hash algorithms are required to support result extension, because (in my opinion) this is extremely useful functionality that is easy - even trivial - to provide, but is often withheld either by accident or in some cases, even deliberately. Hash algorithms typically have a "finalization" phase that pads the message, mixes the length, scrambles the internal state in a more thorough manner than in `update`, and then derives a hash value from that state. (The hash value is often shorter than the total amount of state.) If this "finalization" phase is performed more than once, one naturally gets the mandated `result()` behavior. Falco continues: > I pointed out in the post I already made that the quality of > digest from calling result twice is dependent on the hash > algorithm, and there is no way the library can provide > assurances on the quality That's of course correct, but it also applies to the quality of calling `result()` only once; it's naturally dependent on the implementation of the hash algorithm. What's important here is that it's not possible to provide an extended result of better quality from the outside; the hash algorithm is in the best place to provide it because it has access to more bits of internal state than it lets out. This requirement effectively mandates that all _hash algorithms_ be _extendable-output hash functions_: https://en.wikipedia.org/wiki/Extendable-output_function Note that this is not the only innovation that the proposed hash algorithm concept involves. All hash algorithms are required to support seeding from uint64_t and from an arbitrary sequence of bytes, which makes them effectively _keyed hash functions_ (or _message authentication codes_). Also note that the requirement that one can interleave calls to `update` and `result` arbitrarily makes it possible to implement byte sequence seeding (for algorithms that don't already support it) in the following manner: Hash::Hash( unsigned char const* p, size_t n ): Hash() { if( n != 0 ) { update( p, n ); result(); } } Subsequent `update` calls now start from an initial internal state that has incorporated the contents of [p, p+n), and that has been "finalized" (scrambled thoroughly) such that the result is not equivalent to just prepending the seed to the message (as would have happened if the result() call has been omitted.)

4 14

[hash2] Hash Algorithm Requirements versus concepts
by Andrzej Krzemienski 08 Dec '24

08 Dec '24

Hi Everyone, It strikes me that the description of Hash Algorithm Requirements does not use C++ concepts. Even though the library is C++11-compatible, it could use a concept in the documentation. The goal of the concepts was to serve a case like this, so I wonder if this is just an omission from the authors, or if this is an indication that concepts do not deliver their promise. There is the optional `block_size` but this could be modeled by two concepts HashAlgorithm and BlockHashAlgorithm. The type of `block_size` looks like a candidate for another concept HashAlgorithmResult. Regards, &rzej;

1 0

[hash2][review] Early review (due to holidays)
by Claudio DeSouza 07 Dec '24

07 Dec '24

Hello. My review is the summary of a few discussions I've had with Peter and Christian regarding the API design being employed for hash2. I'm aware hash2 is inspired by one of Hinnat's proposals to standardisation that didn't come to fruition. I would like first to commend the authors for their work on this project. Having good libraries for this type of task is important, especially when the big names like OpenSSL provide not-so-great interfaces. Having said that, I have a few thoughts on what this library may be missing. *Spanification* There's a significant change in mood with regards to safe library design taking place in the industry. With clang, we now have -Wunsafe-buffer-usage, and this warning has been continuously improved recently. For reference https://clang.llvm.org/docs/SafeBuffers.html When I was going through this library's code I was left with a frustrating thought that boost should be at the forefront of this type of innovation: safe, and intuitive library design. Each function that takes a pair of void*/size_t should be a span<uint_8>. The model of how to approach these areas is a solved issue at this point. A robust span type with intuitive extensions would have matching constructors for all sorts of contiguous ranges, including native arrays, which would make using this library much simpler. It would also afford the necessary compile-time validations that you get with static extents in a span. At the current state, hash2 relies on a lot of pinky-promise guarantees that we tend to live with in C++, but that in fact can be avoided without much (or any) overhead. *Digest type* Due to limitations to how much of std::array is constexpr in different C++ versions, this library offers its own digest type. However, this just wraps a stack array. I feel that rather than returning a digest that is stack allocated, it would have been better for algorithms to take a fixed size span where the digest can be written to. I say this because in many cases a stack allocated array will be fine. However a mutable fixed span would provide more flexibility about the storage, for instance, a vector. There's also the matter of hardening (https://libcxx.llvm.org/Hardening.html) which one loses by using this particular digest type, rather than storage provided by a standard type. But I would also like to mention that sha3 has a variable digest size, so again, another reason why an out param mutable span would be a more flexible choice on the whole. *Boost Identity* I have been reluctant to write a review because it seems unfair to demand of hash2 to provide robust spanified interfaces, when no other library before was required to do so. However, at the same time, considering that this type of code tends to be used in safety-sensitive contexts, I fear this library may fail to achieve its potential, when a lot of its interfaces are unsafe. if I had to bring this library to any project I'm working on, I would have to wrap it into safe spanified interfaces, to hide away all these pointer/length pairs around, before I could use it in my code. However, I understand that to produce a spanified design, boost would have first to introduce a mature span implementation + extensions that can be leveraged for this type of code design. *Security* I think for things like HMAC, there is a question of allocators that zero-out memory, of secure obfuscated memory content in case of crash dumps, etc, that this library has not been attentive to, so I thought it was worth mentioning. *In conclusion* I'm here to support the authors. I don't feel like voting either accept or reject, because I think Boost as a whole has not provided the mission critical answer to that. I want this library to succeed. I just wish it was a bit more ambitious in its goal on the library-design space. Claudio.

13 107

[hash2] HashAlgorithm's named requirements for result()
by Vinnie Falco 07 Dec '24

07 Dec '24

On Fri, Dec 6, 2024 at 4:24 PM Ivan Matek via Boost <boost(a)lists.boost.org> wrote: > Goal was to "encourage" users in 2 ways to call result() only once. First > way is that move signals that value is "tainted", second is that clang-tidy > can detect double moves sometimes. > I propose to change HashAlgorithm requirements as follows: --- HashAlgorithm result() is renamed to result_type finalize(); The current documentation for finalize [1] is moved elsewhere and replaced with the following text: This function shall return the final hash value of the input message, where the input message is defined by the ordered sequence of bytes provided in all prior calls to update(). The behavior of subsequent calls to finalize() is undefined unless specified by the HashAlgorithm. --- Rationale: The name "finalize" is closer to established practice and is more suggestive of the typical state mutation: https://github.com/openssl/openssl/blob/5fce85ec52a826d53665552b50e67f86c92… The existing Hash2 documentation for result() is too specific and suggests operations which may not be relevant such as for FNV-1a or other byte-oriented algorithms. It would be better to state only the mandatory requirements and leave the rest to the implementation. The existing Hash2 documentation suggests the possibility of calling finalize() twice or more in a row, yet this can only be considered safe, secure, or otherwise generally following best practices on a case by case basis depending on the HashAlgorithm. The HashAlgorithm concept is first and foremost designed for use with the Hash named requirement (std::hash and related). Making calls to a generic finalize() undefined does not hinder this use-case. There has been no research into whether HashAlgorithm should be held up as the concept for how ALL hash algorithms should be modeled. Therefore the named requirements for HashAlgorithm's finalize() function should only go as far as needed to support the Hash use-case, and we should leave the rest of its behavior up to the implementer. In particular I think there is danger here: template< typename HashAlgorithm > auto double_finalize( HashAlgorithm& h ) { h.finalize(); return h.finalize(); } This is dangerous because the usage in generic contexts results in an unpredictable quality of result. We should use the more strict definition I provided above for now, and only in the future loosen the definition if there is evidence that doing so yields a net benefit. It is always easier to go from strict to loose. And going from loose to strict after the fact is difficult and often impossible without breaking things. It is better if double_finalize is undefined. Users who want random numbers or whatever, can do it with a specific implementation of HashAlgorithm which offers additional guarantees for finalize(). [1] https://pdimov.github.io/hash2/doc/html/hash2.html#hashing_bytes_result

2 2

Any interest in a faster "any" type alternative?
by Marcus Tomlinson 06 Dec '24

06 Dec '24

I’m aware that the current implementation of any intentionally avoids copy assignments and uses only copy construction for all value updates. I believe this is due to the fact that copy assignment isn’t always available for all types - understandable. My proposal is what if we had a separate “fast-any” type that does utilise copy assignment when types match. In my own usage of this new type, I’ve found massive (roughly 2x) performance improvement on consecutive assignments of the same type. I’d like to gauge whether there would be any interest in adding such a type to the Boost library. Source code: https://github.com/cross-platform/fast_any Regards, Marcus Tomlinson

2 2

[release] The Master branch is now closed for ALL changes
by Marshall Clow 05 Dec '24

05 Dec '24

Please do not commit to the master branch. — Marshall

1 0