- Boost - lists.boost.org

[Hash2] Result extension
by Peter Dimov 09 Dec '24

09 Dec '24

Vinnie Falco asked me the following on Slack: > I would ask, what is the motivating use-case for calling result > twice? This is not explained in the docs and no examples are > given. In fact, the one example given says "not to do this" Calling result() twice (or more times) provides result extension; the ability to extract variable number of bits from a hash algorithm, instead of a fixed size value (e.g. 64 bit.) This is in fact stated in the docs here https://pdimov.github.io/hash2/doc/html/hash2.html#hashing_bytes_result > Note that result is non-const, because it changes the internal > state. It’s allowed for result to be called more than once; > subsequent calls perform the state finalization again and as a > result produce a pseudorandom sequence of result_type values. > This can be used to effectively extend the output of the hash > function. For example, a 256 bit result can be obtained from a > hash algorithm whose result_type is 64 bit, by calling result four > times. and there is an example of doing that here https://pdimov.github.io/hash2/doc/html/hash2.html#example_result_extension All hash algorithms are required to support result extension, because (in my opinion) this is extremely useful functionality that is easy - even trivial - to provide, but is often withheld either by accident or in some cases, even deliberately. Hash algorithms typically have a "finalization" phase that pads the message, mixes the length, scrambles the internal state in a more thorough manner than in `update`, and then derives a hash value from that state. (The hash value is often shorter than the total amount of state.) If this "finalization" phase is performed more than once, one naturally gets the mandated `result()` behavior. Falco continues: > I pointed out in the post I already made that the quality of > digest from calling result twice is dependent on the hash > algorithm, and there is no way the library can provide > assurances on the quality That's of course correct, but it also applies to the quality of calling `result()` only once; it's naturally dependent on the implementation of the hash algorithm. What's important here is that it's not possible to provide an extended result of better quality from the outside; the hash algorithm is in the best place to provide it because it has access to more bits of internal state than it lets out. This requirement effectively mandates that all _hash algorithms_ be _extendable-output hash functions_: https://en.wikipedia.org/wiki/Extendable-output_function Note that this is not the only innovation that the proposed hash algorithm concept involves. All hash algorithms are required to support seeding from uint64_t and from an arbitrary sequence of bytes, which makes them effectively _keyed hash functions_ (or _message authentication codes_). Also note that the requirement that one can interleave calls to `update` and `result` arbitrarily makes it possible to implement byte sequence seeding (for algorithms that don't already support it) in the following manner: Hash::Hash( unsigned char const* p, size_t n ): Hash() { if( n != 0 ) { update( p, n ); result(); } } Subsequent `update` calls now start from an initial internal state that has incorporated the contents of [p, p+n), and that has been "finalized" (scrambled thoroughly) such that the result is not equivalent to just prepending the seed to the message (as would have happened if the result() call has been omitted.)

4 14

[hash2] Hash Algorithm Requirements versus concepts
by Andrzej Krzemienski 08 Dec '24

08 Dec '24

Hi Everyone, It strikes me that the description of Hash Algorithm Requirements does not use C++ concepts. Even though the library is C++11-compatible, it could use a concept in the documentation. The goal of the concepts was to serve a case like this, so I wonder if this is just an omission from the authors, or if this is an indication that concepts do not deliver their promise. There is the optional `block_size` but this could be modeled by two concepts HashAlgorithm and BlockHashAlgorithm. The type of `block_size` looks like a candidate for another concept HashAlgorithmResult. Regards, &rzej;

1 0

[hash2][review] Early review (due to holidays)
by Claudio DeSouza 07 Dec '24

07 Dec '24

Hello. My review is the summary of a few discussions I've had with Peter and Christian regarding the API design being employed for hash2. I'm aware hash2 is inspired by one of Hinnat's proposals to standardisation that didn't come to fruition. I would like first to commend the authors for their work on this project. Having good libraries for this type of task is important, especially when the big names like OpenSSL provide not-so-great interfaces. Having said that, I have a few thoughts on what this library may be missing. *Spanification* There's a significant change in mood with regards to safe library design taking place in the industry. With clang, we now have -Wunsafe-buffer-usage, and this warning has been continuously improved recently. For reference https://clang.llvm.org/docs/SafeBuffers.html When I was going through this library's code I was left with a frustrating thought that boost should be at the forefront of this type of innovation: safe, and intuitive library design. Each function that takes a pair of void*/size_t should be a span<uint_8>. The model of how to approach these areas is a solved issue at this point. A robust span type with intuitive extensions would have matching constructors for all sorts of contiguous ranges, including native arrays, which would make using this library much simpler. It would also afford the necessary compile-time validations that you get with static extents in a span. At the current state, hash2 relies on a lot of pinky-promise guarantees that we tend to live with in C++, but that in fact can be avoided without much (or any) overhead. *Digest type* Due to limitations to how much of std::array is constexpr in different C++ versions, this library offers its own digest type. However, this just wraps a stack array. I feel that rather than returning a digest that is stack allocated, it would have been better for algorithms to take a fixed size span where the digest can be written to. I say this because in many cases a stack allocated array will be fine. However a mutable fixed span would provide more flexibility about the storage, for instance, a vector. There's also the matter of hardening (https://libcxx.llvm.org/Hardening.html) which one loses by using this particular digest type, rather than storage provided by a standard type. But I would also like to mention that sha3 has a variable digest size, so again, another reason why an out param mutable span would be a more flexible choice on the whole. *Boost Identity* I have been reluctant to write a review because it seems unfair to demand of hash2 to provide robust spanified interfaces, when no other library before was required to do so. However, at the same time, considering that this type of code tends to be used in safety-sensitive contexts, I fear this library may fail to achieve its potential, when a lot of its interfaces are unsafe. if I had to bring this library to any project I'm working on, I would have to wrap it into safe spanified interfaces, to hide away all these pointer/length pairs around, before I could use it in my code. However, I understand that to produce a spanified design, boost would have first to introduce a mature span implementation + extensions that can be leveraged for this type of code design. *Security* I think for things like HMAC, there is a question of allocators that zero-out memory, of secure obfuscated memory content in case of crash dumps, etc, that this library has not been attentive to, so I thought it was worth mentioning. *In conclusion* I'm here to support the authors. I don't feel like voting either accept or reject, because I think Boost as a whole has not provided the mission critical answer to that. I want this library to succeed. I just wish it was a bit more ambitious in its goal on the library-design space. Claudio.

13 107

[hash2] HashAlgorithm's named requirements for result()
by Vinnie Falco 07 Dec '24

07 Dec '24

On Fri, Dec 6, 2024 at 4:24 PM Ivan Matek via Boost <boost(a)lists.boost.org> wrote: > Goal was to "encourage" users in 2 ways to call result() only once. First > way is that move signals that value is "tainted", second is that clang-tidy > can detect double moves sometimes. > I propose to change HashAlgorithm requirements as follows: --- HashAlgorithm result() is renamed to result_type finalize(); The current documentation for finalize [1] is moved elsewhere and replaced with the following text: This function shall return the final hash value of the input message, where the input message is defined by the ordered sequence of bytes provided in all prior calls to update(). The behavior of subsequent calls to finalize() is undefined unless specified by the HashAlgorithm. --- Rationale: The name "finalize" is closer to established practice and is more suggestive of the typical state mutation: https://github.com/openssl/openssl/blob/5fce85ec52a826d53665552b50e67f86c92… The existing Hash2 documentation for result() is too specific and suggests operations which may not be relevant such as for FNV-1a or other byte-oriented algorithms. It would be better to state only the mandatory requirements and leave the rest to the implementation. The existing Hash2 documentation suggests the possibility of calling finalize() twice or more in a row, yet this can only be considered safe, secure, or otherwise generally following best practices on a case by case basis depending on the HashAlgorithm. The HashAlgorithm concept is first and foremost designed for use with the Hash named requirement (std::hash and related). Making calls to a generic finalize() undefined does not hinder this use-case. There has been no research into whether HashAlgorithm should be held up as the concept for how ALL hash algorithms should be modeled. Therefore the named requirements for HashAlgorithm's finalize() function should only go as far as needed to support the Hash use-case, and we should leave the rest of its behavior up to the implementer. In particular I think there is danger here: template< typename HashAlgorithm > auto double_finalize( HashAlgorithm& h ) { h.finalize(); return h.finalize(); } This is dangerous because the usage in generic contexts results in an unpredictable quality of result. We should use the more strict definition I provided above for now, and only in the future loosen the definition if there is evidence that doing so yields a net benefit. It is always easier to go from strict to loose. And going from loose to strict after the fact is difficult and often impossible without breaking things. It is better if double_finalize is undefined. Users who want random numbers or whatever, can do it with a specific implementation of HashAlgorithm which offers additional guarantees for finalize(). [1] https://pdimov.github.io/hash2/doc/html/hash2.html#hashing_bytes_result

2 2

Any interest in a faster "any" type alternative?
by Marcus Tomlinson 06 Dec '24

06 Dec '24

I’m aware that the current implementation of any intentionally avoids copy assignments and uses only copy construction for all value updates. I believe this is due to the fact that copy assignment isn’t always available for all types - understandable. My proposal is what if we had a separate “fast-any” type that does utilise copy assignment when types match. In my own usage of this new type, I’ve found massive (roughly 2x) performance improvement on consecutive assignments of the same type. I’d like to gauge whether there would be any interest in adding such a type to the Boost library. Source code: https://github.com/cross-platform/fast_any Regards, Marcus Tomlinson

2 2

[release] The Master branch is now closed for ALL changes
by Marshall Clow 05 Dec '24

05 Dec '24

Please do not commit to the master branch. — Marshall

1 0

[release] Master branch is now open for bug fixes
by Ion Gaztañaga 04 Dec '24

04 Dec '24

Hi, The master branch is now open for bug fixes, test and documentation changes, but please follow the policy described in: https://github.com/boostorg/wiki/wiki/Releases%3A-Beta-Merge-Policy The next deadline: On December 4th, master closes for all changes. As always, the calendar is at https://www.boost.org/development/ — The release managers

9 22

[release] Reminder: Master Branch closes for all changes TOMORROW December 4th
by Marshall Clow 03 Dec '24

03 Dec '24

For the 1.87.0 release. The calendar is here: https://www.boost.org/development/index.html — Marshall P.S. If you have change notes, don’t forget to add them to the website.

1 0

Boost 1.87-beta1 still has sha1 failure in boost.compute
by Benjamin Redelings 02 Dec '24

02 Dec '24

Hi, It looks like boost 1.87-beta1 still has the compile failures from version 1.86 in boost::compute::detail::sha1. I see that there are patches from Aug 21, 2024 to address this in the `develop` branch of `boost.compute`. Are these likely to make it into boost 1.87? -BenRI

3 3

[release] Boost 1.87.0 beta 1 released
by Marshall Clow 19 Nov '24

19 Nov '24

Boost release 1.86.0 beta 1 is now available at: https://archives.boost.io/beta/1.87.0.beta1/source/ The SHA256 checksums are: 2e7cdf358526e8937b36b93fc96c501b8d6a0c8436e95e65440de31c48ec3086 boost_1_87_0_b1.tar.bz2 98a86940c13509dfd0288a3075280f46a76c9a3faeb8a9bf49ca7f9e6a271995 boost_1_87_0_b1.tar.gz 2ca4c019d0c96fd160dec5292a7a6bfcf7b4e592cd238a2207e17084f8021d08 boost_1_87_0_b1.zip 5ea32facd963dbb79ceb5bda7ef7546017ea5da854158f8933bbad74d6bbe622 boost_1_87_0_b1.7z This is slightly different than the RC3. Someone committed a documentation fix to master after we had started the release process, and rather than re-do the whole thing (and re-tag everything), we decided to go with the release that we prepared. The person who did this has been informed of the problems that he caused, and I don’t think he’ll do it again. Release notes (still need updates): https://www.boost.org/users/history/version_1_87_x.html Please try the beta and report any problems you encounter. Thanks to everyone who contributed to the release. -- Boost Release Team

4 6