Re: Cryptography for Boost. Boost.Crypto3.

From: Mikhail Komarov (nemo_at_[hidden])
Date: 2020-09-03 19:58:02

It is great to see some positive feedback! Thank you!

Coming to questions:

1. Yes. You are correct. Processing over the stream of various classes convertible to any Integral type (for now, we will generalize that further in some time). Internal architecture about how these streams are being processed is a little bit more complicated (includes Boost.Accumulator usage, caches data with Merkle-Damgard or Sponge constructions etc), but still being concept-based.

Coming to the internal architecture. According to what we’ve been recommended for us to be able to pave the road for this set of concepts to the standard (https://www.reddit.com/r/cpp/comments/ikxgs5/nil_crypto3_conceptbased_pure_c_cryptography/g3ro8dg?utm_source=share&utm_medium=web2x&context=3 <https://www.reddit.com/r/cpp/comments/ikxgs5/nil_crypto3_conceptbased_pure_c_cryptography/g3ro8dg?utm_source=share&utm_medium=web2x&context=3>), the set of concepts used internally will be extended and modified a little bit.

2. Yes. There is even some drafty implementations of such an approach for various cryptography fields in the main =nil; Crypto3 suite (https://github.com/NilFoundation/crypto3 <https://github.com/NilFoundation/crypto3>). They were simply not yet ported to Boost-ified version (https://github.com/NilFoundation/boost-crypto3 <https://github.com/NilFoundation/boost-crypto3>). 

2.1 For example, here is a drafty public key cryptography implementation (including elliptic curves operations, along with signing and verification): https://github.com/NilFoundation/crypto3-pubkey <https://github.com/NilFoundation/crypto3-pubkey>. (work in progress)

2.2 Here is a drafty cryptography-enhanced fork of Boost.Random (https://github.com/NilFoundation/crypto3-random <https://github.com/NilFoundation/crypto3-random>) which uses drafty stream ciphers (https://github.com/NilFoundation/crypto3-stream <https://github.com/NilFoundation/crypto3-stream>) being implemented with the same approach. I would really like to push stream ciphers into Boost.Crypto3 sometime, so after that I could push Boost.Random, enhanced with Salsa/ChaCha RNGs.

2.3 Same architecture is being also applied to drafty message authentication codes in here: https://github.com/NilFoundation/crypto3-mac <https://github.com/NilFoundation/crypto3-mac>. (drafty)

2.4 Password-based key derivation functions: https://github.com/NilFoundation/crypto3-pbkdf <https://github.com/NilFoundation/crypto3-pbkdf> (also drafty)

2.5 Password hashing: https://github.com/NilFoundation/crypto3-passhash <https://github.com/NilFoundation/crypto3-passhash> (also drafty)

2.6 Usual key derivation functions: https://github.com/NilFoundation/crypto3-kdf <https://github.com/NilFoundation/crypto3-kdf> (drafty)

2.7 Cipher modes: https://github.com/NilFoundation/crypto3-modes <https://github.com/NilFoundation/crypto3-modes> (less drafty)

2.8 Codecs: https://github.com/NilFoundation/crypto3-codec <https://github.com/NilFoundation/crypto3-codec> (not drafty)

2.9 And much more things using similar approach mostly listed in here: https://github.com/NilFoundation/crypto3/tree/master/libs <https://github.com/NilFoundation/crypto3/tree/master/libs>.

I would really like to get a clue of what particular fields could be considered useful for Boost, so I could merge them into Boost.Crypto3 and propose for a review afterwards.

Coming to CRC. We have our own implementation of CRC being fit into the general architecture (https://github.com/NilFoundation/crypto3-hash/pull/68 <https://github.com/NilFoundation/crypto3-hash/pull/68>), but the architecture is a little bit too heavy for such small-digested hashes (literally checksums) as CRC or Adler or xxHash or MurmurHash. So that is why I pretty much like the way you got it implemented.

Sincerely yours,

Mikhail Komarov
nemo_at_nil.foundation

> On 3 Sep 2020, at 20:55, Christopher Kormanyos <e_float_at_[hidden]> wrote:
> 
> > By the way.
> > The code: https://github.com/NilFoundation/boost-crypto3%c2  <https://github.com/NilFoundation/boost-crypto3><https://github.com/NilFoundation/boost-crypto3 <https://github.com/NilFoundation/boost-crypto3>>.
> 
> I really like this design. Having done something
> very similar (probably not-to-be published),
> I appreciate very much the true resource-clarity
> in its design. In some brief glances at the code,
> I see it is favoring compile-time sizes over
> dynamic allocation via use of <array>,
> <type_traits>, etc. Fast and clean!
> There is sensible use of constexpr too.
> Honeslty, this looks like an attempt at crypto
> that potentially lends itself well to PC as well
> as the resource-constrained world of bare-metal
> alike. Nice.
> 
> Questions: Am I correct in noting that
> hashing and block ciphers in this design
> are seen as generalized stream transformations?
> If so, do you have thoughts about how (or if)
> this design extends to digital signatures
> such as ECC having both signature and
> signature verification?
> Are random generators such as Fortuna,
> Salsa or ChaCha also stream transformations
> in this paradigm?
> 
> 
> > Cristopher (Kormanyos), I’m aware of your
> > CRC catalog library (https://github.com/ckormanyos/crc-catalog%c2  <https://github.com/ckormanyos/crc-catalog><https://github.com/ckormanyos/crc-catalog <https://github.com/ckormanyos/crc-catalog>>)
> > I like it because of how in particular it is implemented.
> > I would really like to merge it into our CRC
> > implementation in Crypto3.Hash.
> 
> Thank you. In its present state, however,
> there's only bitwise calculations.
> The catalog, as implemented, does utilize
> a clear, generic, stream-oriented approach
> which would fit with your overall work.
> But if you need table-driven methods
> or table generation, we would need to
> discuss these.
> 
> Thank you for sharing this work.
> 
> Kind regards, Chris
> On Wednesday, September 2, 2020, 2:52:42 AM GMT+2, Mikhail Komarov via Boost <boost_at_[hidden] <mailto:boost_at_[hidden]>> wrote:
> 
> 
> Hello!
> 
> Some time ago I promised to show something about cryptography library architecture and implementation for Boost  (https://lists.boost.org/Archives/boost//2020/02/248205.php%c2  <https://lists.boost.org/Archives/boost//2020/02/248205.php><https://lists.boost.org/Archives/boost//2020/02/248205.php <https://lists.boost.org/Archives/boost//2020/02/248205.php>>), so here I am. I’m actually glad my two innocent letters triggered such a conversation. That is what I wanted to see!
> 
> Please, read this letter very carefully, because some explaining needs to be done before you go to the code.
> 
> By the way. The code: https://github.com/NilFoundation/boost-crypto3%c2  <https://github.com/NilFoundation/boost-crypto3><https://github.com/NilFoundation/boost-crypto3 <https://github.com/NilFoundation/boost-crypto3>>.
> 
> Warning. This letter contains mentioning of my non-commercial organizations I develop Boost.Crypto3 with. Don’t consider that as advertising. It is not supposed to be an advertisement.
> 
> So, what cryptography for Boost (and probably STL some time after) should look like?
> 
> 1. It should definitely be simple to use and implemented with some generic programming spirit.
> 
> That is exactly what was done. Remember std::transform? 
> Typical usage technique for all of the modules of a suite looks as follows.
> 
> Classic iterator pair with output iterator:
> 
> using namespace boost::crypto3;
> std::string input = “Weird German 2 byte thing: Ã.“, out;
> encode<codec::base64>(input.begin(), input.end(), std::inserter(out, out.end()));
> 
> Classic iterator pair with output value return:
> 
> using namespace boost::crypto3;
> std::string input = “00112233445566778899aabbccddeeff”;
> std::string key = “000102030405060708090a0b0c0d0e0f”;
> std::string out = encrypt<block::rijndael<128, 128>>(input.begin(), input.end(), key.begin(), key.end());
> 
> C++20 Ranges (or Boost.Range) usage:
> 
> using namespace boost::crypto3;
> std::vector<std::uint8_t> input = {0x27, 0x0f, 0xb1, 0x89, 0x82, 0x80, 0x0d, 0xa6, 0x40};
> std::string out = encode<codec::base32>(input);
> // or
> encode<codec::base32>(input, out);
> 
> What about generic encoding/encryption/signing/hashing/%name_your_action% interface for classes?
> 
> You can definitely do that in case you define implicit conversion of your class to integral type. I’m thinking of introducing Inspectable concept which will require for the developer to have a function in his class which would return every class member which developer thinks needs to be encrypted/hashes/encoded/etc, converted to the particular algorithm’s format. (e.g. satisfying Integral concept for ciphers/codes/hashes).
> 
> 2. It should contain only time-proven schemes implemented with well-known and proven techniques. 
> 
> Yes. Boost.Crypto3 (https://github.com/NilFoundation/boost-crypto3%c2  <https://github.com/NilFoundation/boost-crypto3><https://github.com/NilFoundation/boost-crypto3 <https://github.com/NilFoundation/boost-crypto3>>) is derived from =nil; Crypto3 C++ Cryptography Suite (https://github.com/NilFoundation/crypto3%c2  <https://github.com/NilFoundation/crypto3><https://github.com/NilFoundation/crypto3 <https://github.com/NilFoundation/crypto3>>), which my non-commercial foundation designed and developed for it’s projects.
> 
> A little bit of information on how exactly it was derived from the original suite you can get from the readme.md in both or repositories.
> 
> As far as you can see in the original repository, the suite contains some very novel schemes 
> (e.g. Verifiable Delay Functions: https://github.com/nilfoundaton/crypto3-vdf.git%c2  <https://github.com/nilfoundaton/crypto3-vdf.git><https://github.com/nilfoundaton/crypto3-vdf.git <https://github.com/nilfoundaton/crypto3-vdf.git>>) and it is going to contain more (threshold cryptography, zero-knowlege 
> cryptography, cryptographic accumulators etc.). But Boost-ified version has to be a solid one. No novel
> schemes. Only proven ones with proven implementation techniques. Which ones? This is what Boost community is up to decide.
> 
> So, the question for the community. What schemes should be included?
> 
> For now we only present small set of block ciphers, hashes and codecs listed in here https://crypto3.nil.foundation/projects/crypto3/db/d97/group__codec.html%c2  <https://crypto3.nil.foundation/projects/crypto3/db/d97/group__codec.html><https://crypto3.nil.foundation/projects/crypto3/db/d97/group__codec.html <https://crypto3.nil.foundation/projects/crypto3/db/d97/group__codec.html>>, https://crypto3.nil.foundation/projects/crypto3/db/d97/group__hashes.html%c2  <https://crypto3.nil.foundation/projects/crypto3/db/d97/group__hashes.html><https://crypto3.nil.foundation/projects/crypto3/db/d97/group__hashes.html <https://crypto3.nil.foundation/projects/crypto3/db/d97/group__hashes.html>>, https://crypto3.nil.foundation/projects/crypto3/d2/dba/group__block.html%c2  <https://crypto3.nil.foundation/projects/crypto3/d2/dba/group__block.html><https://crypto3.nil.foundation/projects/crypto3/d2/dba/group__block.html <https://crypto3.nil.foundation/projects/crypto3/d2/dba/group__block.html>>. (Or you can find the list in the include directories of particular repositories: https://github.com/nilfoundation/crypto3-block.git%c2  <https://github.com/nilfoundation/crypto3-block.git><https://github.com/nilfoundation/crypto3-block.git <https://github.com/nilfoundation/crypto3-block.git>>, https://github.com/nilfoundation/crypto3-codec%c2  <https://github.com/nilfoundation/crypto3-codec><https://github.com/nilfoundation/crypto3-codec.git <https://github.com/nilfoundation/crypto3-codec.git>>.git <https://github.com/nilfoundation/crypto3-codec.git <https://github.com/nilfoundation/crypto3-codec.git>>, https://github.com/nilfoundation/crypto3-hash%c2  <https://github.com/nilfoundation/crypto3-hash><https://github.com/nilfoundation/crypto3-hash.git <https://github.com/nilfoundation/crypto3-hash.git>>.git <https://github.com/nilfoundation/crypto3-hash.git <https://github.com/nilfoundation/crypto3-hash.git>>).
> 
> Peter (Dimov), I’m aware of your abandoned hash library implementation (https://github.com/pdimov/hash2%c2  <https://github.com/pdimov/hash2><https://github.com/pdimov/hash2 <https://github.com/pdimov/hash2>>) based on Vinnie’s paper. I would really like to see it’s contents and concepts (probably) adapted to hash module of Boost.Crypto3. Would be a great synergy.
> 
> 3. It should have a concept-based architecture.
> 
> Concept-based architecture supposes every reused type trait to be formalized, and standardized. This is what exactly has been done. In particular, concepts for contents of Boost.Crypto3 are listed in sufficient documentation chapters: https://crypto3.nil.foundation/projects/crypto3/df/d5d/block_ciphers_concepts.html%c2  <https://crypto3.nil.foundation/projects/crypto3/df/d5d/block_ciphers_concepts.html><https://crypto3.nil.foundation/projects/crypto3/df/d5d/block_ciphers_concepts.html <https://crypto3.nil.foundation/projects/crypto3/df/d5d/block_ciphers_concepts.html>>, https://crypto3.nil.foundation/projects/crypto3/d7/da8/codec_concepts.html%c2  <https://crypto3.nil.foundation/projects/crypto3/d7/da8/codec_concepts.html><https://crypto3.nil.foundation/projects/crypto3/d7/da8/codec_concepts.html <https://crypto3.nil.foundation/projects/crypto3/d7/da8/codec_concepts.html>>, https://crypto3.nil.foundation/projects/crypto3/d1/d1f/hashes_concepts.html%c2  <https://crypto3.nil.foundation/projects/crypto3/d1/d1f/hashes_concepts.html><https://crypto3.nil.foundation/projects/crypto3/d1/d1f/hashes_concepts.html <https://crypto3.nil.foundation/projects/crypto3/d1/d1f/hashes_concepts.html>>.
> 
> Long story short. We standardize stateless policies containing all the params as constexpr ones used for functions (interface-functions like encrypt/decrypt) and actual algorithms.
> 
> We also figured out that most of cryptographic algorithms are about accumulation and only then processing. So we used Accumulator concept from Boost.Accumulator to accumulate (surprise surprise) data in a format, which is required by particular algorithm to be passed into.
> 
> You can get the idea of how I’ve merged the Accumulator concept in from architecture and implementation notes (for Block Ciphers) in here: https://crypto3.nil.foundation/projects/crypto3/db/dee/block_ciphers_impl.html%c2  <https://crypto3.nil.foundation/projects/crypto3/db/dee/block_ciphers_impl.html><https://crypto3.nil.foundation/projects/crypto3/db/dee/block_ciphers_impl.html <https://crypto3.nil.foundation/projects/crypto3/db/dee/block_ciphers_impl.html>>. Pretty drafty as well, but better than nothing.
> 
> Yes, Hash concept should be made consent with currently existing Hash concept. This is in progress.
> 
> Vinnie (Falco), I remember, you’ve mentioned some more generic concepts like OneWayFunction (in here: https://lists.boost.org/Archives/boost//2020/02/248228.php%c2  <https://lists.boost.org/Archives/boost//2020/02/248228.php><https://lists.boost.org/Archives/boost//2020/02/248228.php <https://lists.boost.org/Archives/boost//2020/02/248228.php>>). I agree, we should do that, but we simply had no time to put this fine idea to our architecture. We would really appreciate any help with that.
> 
> Cristopher (Kormanyos), I’m aware of your CRC catalog library (https://github.com/ckormanyos/crc-catalog%c2  <https://github.com/ckormanyos/crc-catalog><https://github.com/ckormanyos/crc-catalog <https://github.com/ckormanyos/crc-catalog>>) I like it because of how in particular it is implemented. I would really like to merge it into our CRC implementation in Crypto3.Hash.
> 
> By the way. I wonder if we could sometime remove separate Boost.CRC library and use CRC from more generic Boost.Crypto3? Hypothetically. Just asking.
> 
> Yeah, we have checksums being implemented in Crypto3.Hash (and in Boost.Crypto3 as well), but they were simply not verified yet, so they are hanging in a PR (https://github.com/NilFoundation/crypto3-hash/pull/68%c2  <https://github.com/NilFoundation/crypto3-hash/pull/68><https://github.com/NilFoundation/crypto3-hash/pull/68 <https://github.com/NilFoundation/crypto3-hash/pull/68>>).
> 
> 4. Other modules? Roadmap?
> 
> Yes, there will be other modules being proposed in some time. Public Key cryptography in Boost? Yes. Based on enhanced for cryptographic purposes Boost.Multiprecision (https://github.com/NilFoundation/crypto3-multiprecision%c2  <https://github.com/NilFoundation/crypto3-multiprecision><https://github.com/NilFoundation/crypto3-multiprecision <https://github.com/NilFoundation/crypto3-multiprecision>>). SSL based on Boost.Crypto3? Yes. Why not. We are already in progress.
> 
> Major updates are going to be performed with merging in the Boost-ified version of bunch of cross-dependent modules of the original suite. For example, public key cryptography would require enhancing Boost.Multiprecision (which is already in progress) and introducing finite fields library (https://github.com/NilFoundation/crypto3-algebra%c2  <https://github.com/NilFoundation/crypto3-algebra><https://github.com/NilFoundation/crypto3-algebra <https://github.com/NilFoundation/crypto3-algebra>>).
> 
> By the way. The original =nil; Crypto3 suite is highly modular. Every cryptography field is emplaced in separate submodule. There are more than 30 of them now in total (counting closed ones)(yeah, it is huge). Would you like to keep this modularity (like introducing Boost.BlockCiphers, Boost.Hashes and Boost.Codec instead of Boost.Crypto3)? Or do we need to keep everything in header-only Boost.Crypto3? I would vote for the second option.
> 
> John (Maddock), do you understand now why I was attempting to get accepted  those strange PRs in Boost.Multiprecision for the last year?
> 
> Degski (Sorry, I’m not aware of your name), I remember you’ve mentioned some kind of compression library of yours. I would like to see the compression in the  suite. Do you think I could fork it and fit into our architecture?
> 
> 5. Why do we think we can implement properly design and cryptography? Ask me :).
> 
> 6. Conclusion.
> 
> In some time, this would definitely result in C++ Standardization Committee proposal. It is a long way, but I hope to bring it there.
> 
> What we would like to hear from Boost community is some guidance about what in particular from this field would be useful for Boost, proposals and suggestions about architecture and concepts. 
> 
> Let us standardize some cryptography together (starting with this).
> 
> Sincerely yours,
> 
> Mikhail Komarov
> nemo_at_nil.foundation <mailto:nemo_at_nil.foundation>
> 
> 
> _______________________________________________
> Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost <http://lists.boost.org/mailman/listinfo.cgi/boost>