- Boost - lists.boost.org

buffer_size behavior in boost::iostreams::filtering_istream
by Justin McManus 07 Sep '20

07 Sep '20

I have some code that works as intended, but it requires setting a buffer_size parameter to zero on a std::ifstream pushed onto a filtering chain, and I'd like to understand why, to ensure I'm not introducing a bug or a hack. I have essentially the following code: -------------------------------------------------------------------------------------------------------- std::ifstream m_jf("json_filename", std::ios_base::in | std::ios_base::binary); std::locale utf8_locale("en_US.UTF-8"); m_jf.imbue(utf8_locale); boost::iostreams::filtering_istream m_inbuf; m_inbuf.push(boost::iostreams::bzip2_decompressor()); m_inbuf.push(m_jf); std::string m_line; while (std::getline(m_inbuf, m_line)) { // Process the current line from the JSON file } -------------------------------------------------------------------------------------------------------- What I find is that the std::getline call will fail before the code has reached the EOF. It will always fail at the same line in a given JSON file, but it will fail on different lines in different JSON files. It's perfectly reproducible. However, if I change lines 4 and 5 to m_inbuf.push(boost::iostreams::bzip2_decompressor(), *0*); m_inbuf.push(m_jf, *0*); then the problem goes away. My question is, Why does setting the buffer_size parameter to zero solve the issue? What does this do, exactly? I saw the suggestion to set the buffer size this way from an old post in 2009, and it appears to work, but I'd like a deeper understanding of what's happening under the hood. If the buffer size is set to zero, what does the underlying implementation do, and how might this influence whether std::getline fails before the EOF? Thanks very much, Justin -- Justin McManus, Ph.D. Principal Scientist Lead Computational Biologist and Statistical Geneticist Kallyope, Inc. 430 East 29th Street, Suite 1050 New York, NY 10016

3 4

type detection macros + multiprecision + io
by Gero Peterhoff 07 Sep '20

07 Sep '20

Hello Developer, I noticed that some type detection macros and boost::float128_type or numeric_limits do not work reliably: - BOOST_HAS_FLOAT128 with clang and icc do not work, gcc is ok - BOOST_HAS_CHAR32/16_T, BOOST_NO_CHAR32/16_T are not defined, but either BOOST_HAS_TYPE or BOOST_NO_TYPE should be defined; see later - with clang there is no boost::float128_type and an error when including boost/multiprecision/float128.hpp (quadmath.h) - with clang and icc the numeric_limits<float128_t> are not available (gcc ok) see https://godbolt.org/z/95PfTG In general it would make sense: - convert the detection macros to a uniform naming scheme: always BOOST_HAS_TYPE_T (with _T at the end) or BOOST_HAS_TYPE (without _T), but not mixed/both - in multiprecision and math: 1) float: renaming typenames to eg floatBits_mbt (m: multiprecision, b: binary); floatBits_mdt (d: decimal) 2) integer: renaming typenames to e.g. u/intBits_mt; u/intBits_mct (c: checked) 3) based on this, renaming boost::float/int128_type to boost::float/int128_t so that it is always clear and there is no confusion. You can do that for all types, e.g. some DSP/microcontrollers do not support u/int8/64_t etc. I did that quickly for gcc (has_type.hpp) - of course that has to be in the corresponding boost/config/compiler/* and platform/* headers are implemented. Building on this, feature test macros can then be made available (features.hpp). Can you please implement the io-operators and string-conversions for u/int128 (in e.g. boost/math/cstdint/cstdint_iostream.hpp)? thx and regards Gero

5 4

[sync] Documentation ?
by Edward Diener 06 Sep '20

06 Sep '20

There is a 'sync' library in the Boost build tree as a submodule, but I see no documentation for it in Boost 1.74 itself. Is this currently a Boost library, or what is it ?

2 1

Cryptography for Boost. Boost.Crypto3.
by Mikhail Komarov 04 Sep '20

04 Sep '20

Hello! Some time ago I promised to show something about cryptography library architecture and implementation for Boost (https://lists.boost.org/Archives/boost//2020/02/248205.php <https://lists.boost.org/Archives/boost//2020/02/248205.php>), so here I am. I’m actually glad my two innocent letters triggered such a conversation. That is what I wanted to see! Please, read this letter very carefully, because some explaining needs to be done before you go to the code. By the way. The code: https://github.com/NilFoundation/boost-crypto3 <https://github.com/NilFoundation/boost-crypto3>. Warning. This letter contains mentioning of my non-commercial organizations I develop Boost.Crypto3 with. Don’t consider that as advertising. It is not supposed to be an advertisement. So, what cryptography for Boost (and probably STL some time after) should look like? 1. It should definitely be simple to use and implemented with some generic programming spirit. That is exactly what was done. Remember std::transform? Typical usage technique for all of the modules of a suite looks as follows. Classic iterator pair with output iterator: using namespace boost::crypto3; std::string input = “Weird German 2 byte thing: Ã.“, out; encode<codec::base64>(input.begin(), input.end(), std::inserter(out, out.end())); Classic iterator pair with output value return: using namespace boost::crypto3; std::string input = “00112233445566778899aabbccddeeff”; std::string key = “000102030405060708090a0b0c0d0e0f”; std::string out = encrypt<block::rijndael<128, 128>>(input.begin(), input.end(), key.begin(), key.end()); C++20 Ranges (or Boost.Range) usage: using namespace boost::crypto3; std::vector<std::uint8_t> input = {0x27, 0x0f, 0xb1, 0x89, 0x82, 0x80, 0x0d, 0xa6, 0x40}; std::string out = encode<codec::base32>(input); // or encode<codec::base32>(input, out); What about generic encoding/encryption/signing/hashing/%name_your_action% interface for classes? You can definitely do that in case you define implicit conversion of your class to integral type. I’m thinking of introducing Inspectable concept which will require for the developer to have a function in his class which would return every class member which developer thinks needs to be encrypted/hashes/encoded/etc, converted to the particular algorithm’s format. (e.g. satisfying Integral concept for ciphers/codes/hashes). 2. It should contain only time-proven schemes implemented with well-known and proven techniques. Yes. Boost.Crypto3 (https://github.com/NilFoundation/boost-crypto3 <https://github.com/NilFoundation/boost-crypto3>) is derived from =nil; Crypto3 C++ Cryptography Suite (https://github.com/NilFoundation/crypto3 <https://github.com/NilFoundation/crypto3>), which my non-commercial foundation designed and developed for it’s projects. A little bit of information on how exactly it was derived from the original suite you can get from the readme.md in both or repositories. As far as you can see in the original repository, the suite contains some very novel schemes (e.g. Verifiable Delay Functions: https://github.com/nilfoundaton/crypto3-vdf.git <https://github.com/nilfoundaton/crypto3-vdf.git>) and it is going to contain more (threshold cryptography, zero-knowlege cryptography, cryptographic accumulators etc.). But Boost-ified version has to be a solid one. No novel schemes. Only proven ones with proven implementation techniques. Which ones? This is what Boost community is up to decide. So, the question for the community. What schemes should be included? For now we only present small set of block ciphers, hashes and codecs listed in here https://crypto3.nil.foundation/projects/crypto3/db/d97/group__codec.html <https://crypto3.nil.foundation/projects/crypto3/db/d97/group__codec.html>, https://crypto3.nil.foundation/projects/crypto3/db/d97/group__hashes.html <https://crypto3.nil.foundation/projects/crypto3/db/d97/group__hashes.html>, https://crypto3.nil.foundation/projects/crypto3/d2/dba/group__block.html <https://crypto3.nil.foundation/projects/crypto3/d2/dba/group__block.html>. (Or you can find the list in the include directories of particular repositories: https://github.com/nilfoundation/crypto3-block.git <https://github.com/nilfoundation/crypto3-block.git>, https://github.com/nilfoundation/crypto3-codec <https://github.com/nilfoundation/crypto3-codec.git>.git <https://github.com/nilfoundation/crypto3-codec.git>, https://github.com/nilfoundation/crypto3-hash <https://github.com/nilfoundation/crypto3-hash.git>.git <https://github.com/nilfoundation/crypto3-hash.git>). Peter (Dimov), I’m aware of your abandoned hash library implementation (https://github.com/pdimov/hash2 <https://github.com/pdimov/hash2>) based on Vinnie’s paper. I would really like to see it’s contents and concepts (probably) adapted to hash module of Boost.Crypto3. Would be a great synergy. 3. It should have a concept-based architecture. Concept-based architecture supposes every reused type trait to be formalized, and standardized. This is what exactly has been done. In particular, concepts for contents of Boost.Crypto3 are listed in sufficient documentation chapters: https://crypto3.nil.foundation/projects/crypto3/df/d5d/block_ciphers_concep… <https://crypto3.nil.foundation/projects/crypto3/df/d5d/block_ciphers_concep…>, https://crypto3.nil.foundation/projects/crypto3/d7/da8/codec_concepts.html <https://crypto3.nil.foundation/projects/crypto3/d7/da8/codec_concepts.html>, https://crypto3.nil.foundation/projects/crypto3/d1/d1f/hashes_concepts.html <https://crypto3.nil.foundation/projects/crypto3/d1/d1f/hashes_concepts.html>. Long story short. We standardize stateless policies containing all the params as constexpr ones used for functions (interface-functions like encrypt/decrypt) and actual algorithms. We also figured out that most of cryptographic algorithms are about accumulation and only then processing. So we used Accumulator concept from Boost.Accumulator to accumulate (surprise surprise) data in a format, which is required by particular algorithm to be passed into. You can get the idea of how I’ve merged the Accumulator concept in from architecture and implementation notes (for Block Ciphers) in here: https://crypto3.nil.foundation/projects/crypto3/db/dee/block_ciphers_impl.h… <https://crypto3.nil.foundation/projects/crypto3/db/dee/block_ciphers_impl.h…>. Pretty drafty as well, but better than nothing. Yes, Hash concept should be made consent with currently existing Hash concept. This is in progress. Vinnie (Falco), I remember, you’ve mentioned some more generic concepts like OneWayFunction (in here: https://lists.boost.org/Archives/boost//2020/02/248228.php <https://lists.boost.org/Archives/boost//2020/02/248228.php>). I agree, we should do that, but we simply had no time to put this fine idea to our architecture. We would really appreciate any help with that. Cristopher (Kormanyos), I’m aware of your CRC catalog library (https://github.com/ckormanyos/crc-catalog <https://github.com/ckormanyos/crc-catalog>) I like it because of how in particular it is implemented. I would really like to merge it into our CRC implementation in Crypto3.Hash. By the way. I wonder if we could sometime remove separate Boost.CRC library and use CRC from more generic Boost.Crypto3? Hypothetically. Just asking. Yeah, we have checksums being implemented in Crypto3.Hash (and in Boost.Crypto3 as well), but they were simply not verified yet, so they are hanging in a PR (https://github.com/NilFoundation/crypto3-hash/pull/68 <https://github.com/NilFoundation/crypto3-hash/pull/68>). 4. Other modules? Roadmap? Yes, there will be other modules being proposed in some time. Public Key cryptography in Boost? Yes. Based on enhanced for cryptographic purposes Boost.Multiprecision (https://github.com/NilFoundation/crypto3-multiprecision <https://github.com/NilFoundation/crypto3-multiprecision>). SSL based on Boost.Crypto3? Yes. Why not. We are already in progress. Major updates are going to be performed with merging in the Boost-ified version of bunch of cross-dependent modules of the original suite. For example, public key cryptography would require enhancing Boost.Multiprecision (which is already in progress) and introducing finite fields library (https://github.com/NilFoundation/crypto3-algebra <https://github.com/NilFoundation/crypto3-algebra>). By the way. The original =nil; Crypto3 suite is highly modular. Every cryptography field is emplaced in separate submodule. There are more than 30 of them now in total (counting closed ones)(yeah, it is huge). Would you like to keep this modularity (like introducing Boost.BlockCiphers, Boost.Hashes and Boost.Codec instead of Boost.Crypto3)? Or do we need to keep everything in header-only Boost.Crypto3? I would vote for the second option. John (Maddock), do you understand now why I was attempting to get accepted those strange PRs in Boost.Multiprecision for the last year? Degski (Sorry, I’m not aware of your name), I remember you’ve mentioned some kind of compression library of yours. I would like to see the compression in the suite. Do you think I could fork it and fit into our architecture? 5. Why do we think we can implement properly design and cryptography? Ask me :). 6. Conclusion. In some time, this would definitely result in C++ Standardization Committee proposal. It is a long way, but I hope to bring it there. What we would like to hear from Boost community is some guidance about what in particular from this field would be useful for Boost, proposals and suggestions about architecture and concepts. Let us standardize some cryptography together (starting with this). Sincerely yours, Mikhail Komarov nemo(a)nil.foundation

6 8

Re: [boost] ASIO SSL Memory Leaks
by abrarov＠mail.ru 01 Sep '20

01 Sep '20

Hi John, Welcome to this mailing list! Doesn't Asio call SSL_shutdown here: https://github.com/chriskohlhoff/asio/blob/master/asio/include/asio/ssl/deta il/impl/engine.ipp#L325 ? Regarding memory leaks - cold you please provide a minimal example reproducing memory leak including the way you detect memory leak? Could chriskohlhoff/asio GitHub issue #368 (https://github.com/chriskohlhoff/asio/issues/368) be your case? Regards, Marat Abrarov.

1 0

ASIO SSL Memory Leaks
by John A 01 Sep '20

01 Sep '20

Hello y'all, This is my first post to this listserv though I've been on others for ~30 years. I have a Boost question, ob duh. Using the SSL_shutdown() outlined here: https://www.openssl.org/docs/man1.0.2/man3/SSL_shutdown.html from within Boost, there isn't an object to associate with this shutdown call. Secondly, we have memory leaks (~10k bytes) using the SSL library via Boost socket calls using the ASIO SSL streams. What am I missing? What advisement can be offered? Ideally, we want to call SSL_shutdown() from within Boost, though I am missing something. Best regards, John Aronetics.com We Speak ITR +1-216/307-5760

3 3

ASIO: Same code for tcp, unix domain sockets (websockets via beast)
by Matthias Loy 26 Aug '20

26 Aug '20

Hello, we are implementing a service that is to be available via tcp, unix domain sockets (and websocket). Websocket might be out of scope since it is not part of asio. All those protocols are streaming protocols and should behave in the same way: read some bytes, write some bytes. I would like to have the code implemented only once to handle several protocols. Not having one for local::stream_protocol::socket and another one for ip::tcp::socket. Until now I was not able to find something like base classes that allow me to do that. I only found code that works for tcp or unix domain sockets (local). The protocol is not to be decided at compile time but on runtime. Hence I'm having problems using templates here. The documentation of async_read states that the stream has to support the AsyncReadStream concept. Is there a way to get something supporting this concept from local::stream_protocol::socket and ip::tcp::socket? Best regards, Matthias

4 3

JSON library scheduled for review
by Mateusz Loskot 25 Aug '20

25 Aug '20

Hi, A brief announcement from the review wizards: Formal review of the JSON library has been scheduled to take place between October 9 and October 18, 2020. Details and links available here https://www.boost.org/community/review_schedule.html We'd like to thank Pranam Lashkari (Boost.GIL, [Boost.]Astronomy, Boost GSoC) for volunteering to act as the review manager. Best regards, -- Mateusz Loskot, http://mateusz.loskot.net

2 2

Shallow Hal Needs A Gal!
by Vinnie Falco 25 Aug '20

25 Aug '20

And Boost.JSON needs a review manager! -- Regards, Vinnie Follow me on GitHub: https://github.com/vinniefalco

3 4

Modules
by Damian Vicino 24 Aug '20

24 Aug '20

Hi, Are there any guidelines on the use of modules inside boost? Are we planning any steps for adoption of them?

9 18