April 2019 - Boost - lists.boost.org

undefined behavior in Boost.Program_Options
by Chris Rorvick 25 Apr '19

25 Apr '19

I am seeing my GCC ubsan builds littered with "downcast of address xxxxxxxx which does not point to an object of type 'holder'" errors [3] coming from Boost.Program_options (Boost 1.69). The issue has been documented in various places [1][2] but I do not see anything on the mailing list in recent months; apologies if I have missed something. A workaround is proposed on the StackOverflow post [1]. Does anyone have any thoughts on this? Many thanks! Chris [1] https://stackoverflow.com/questions/38987728/why-does-boostany-exhibit-unde… [2] https://github.com/boostorg/program_options/issues/76 [3] .../boost/boost/any.hpp:249:17: runtime error: downcast of address 0x602000053610 which does not point to an object of type 'holder' 0x602000053610: note: object is of type 'boost::any::holder<std::string>' 6f 00 80 67 d0 91 ec 88 fe 7f 00 00 68 33 09 00 40 60 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ^~~~~~~~~~~~~~~~~~~~~~~ vptr for 'boost::any::holder<std::string>' #0 0x7ffe8df13e3d in boost::program_options::typed_value<std::vector<std::string, std::allocator<std::string> >, char>::xparse(boost::any&, std::vector<std::string, std::allocator<std::string> > const&) const (redacted+0x28be3d) #1 0x7ffe88bb0cd5 in boost::program_options::value_semantic_codecvt_helper<char>::parse(boost::any&, std::vector<std::string, std::allocator<std::string> > const&, bool) const (.../libboost_program_options.so.1.69.0+0xdacd5) #2 0x7ffe88ba9193 in boost::program_options::store(boost::program_options::basic_parsed_options<char> const&, boost::program_options::variables_map&, bool) (.../libboost_program_options.so.1.69.0+0xd3193) #3 0x7fff0b727dbf in rdx::launch::OptionsWrapper<boost::program_options::options_description, boost::program_options::variables_map>::prepare(int, char**, std::vector<int (*)(boost::program_options::options_description&), std::allocator<int (*)(boost::program_options::options_description&)> > const&) (redacted+0x448dbf) #4 0x7fff0b755e80 in int generic_wrapper<std::vector<int (*)(boost::program_options::options_description&), std::allocator<int (*)(boost::program_options::options_description&)> >, rdx::Function<int (boost::program_options::variables_map const&), 56u>, rdx::launch::OptionsWrapper<boost::program_options::options_description, boost::program_options::variables_map> >(int, char**, std::vector<int (*)(boost::program_options::options_description&), std::allocator<int (*)(boost::program_options::options_description&)> > const&, rdx::Function<int (boost::program_options::variables_map const&), 56u> const&, rdx::launch::OptionsWrapper<boost::program_options::options_description, boost::program_options::variables_map>&) (redacted+0x476e80) #5 0x7fff0b76593c in redacted(std::string const&, int, char**, std::vector<int (*)(boost::program_options::options_description&), std::allocator<int (*)(boost::program_options::options_description&)> > const&, rdx::Function<int (boost::program_options::variables_map const&), 56u> const&) (redacted+0x48693c) #6 0x95c21d in main (redacted+0x95c21d) #7 0x7ffe82fb83d4 in __libc_start_main (/lib64/libc.so.6+0x223d4) #8 0x963523 (redacted+0x963523)

3 2

Re: [boost] CMake, modular Boost, and other stories
by Mike 24 Apr '19

24 Apr '19

> Hmmm - so no boost library should include any > other boost library? DoesÂ that seem a good idea? Please don't put words in my mouth. I didn't say that and you know it. But the dependency graph could easily be significantly simplified once c++03 is no longer a concern. Again: I'm talking about the future, not the current state. Just an example: [1]Boost.TypeErasure depends on [2]Boost.Thread(and all that entails), just because somewhere in its bowels it needs a mutex. > It's always been good practice to notÂ repeat/copy > functionality in one library into another. Not always. A recent example I stumbled over: Boost.Signal2 depended on boost multi index (which itself depends on a lot of other libs) because it reused it's scope guard implementation (IIRC about 50 lines of straight forward code). Tying yourself to another library for something trivial like that should not be a no-brainer. > But what is really needed - and it may already exist > in BCP but hasn'tÂ been promoted is a simple tool: > a) start with one's application > b) run the app through a tool which recurrsively > reads headers > c) and creates a list of all headers needed > d) extract library names from that list > e) add the git repo of those libraries to my apps > project. So boost should not only get its unique build system, but also its unique dependency management tool? I can tell you as a user: Thanks but no thanks. Also: Didn't you forget source files somewhere in there? What about tests? Bottom line: If boost as a whole was developed as a single project with some coordinated devlopment goal and strategy , it would make sense to distribute it as a single package/framework and a single cmake config file. It seems to me however, that it is more a (huge) collection of independently developed libraries, so I think it is more appropriate to treat them as such, even if there is a huge amount of coupling between them. Am 23.04.19, 23:56, Robert Ramey via Boost <boost(a)lists.boost.org> schrieb: On 4/23/19 2:28 PM, Mike via Boost wrote: > I for one very much hope that boost (at least the libraries that > are actively maintained) will become less coupled > in the future, such that when I just want to use > library X, the rest of boost doesnʼt come as a > dependency with it. Hmmm - so no boost library should include any other boost library? Does that seem a good idea? Itʼs always been good practice to not repeat/copy functionality in one library into another. Iʼve said it many times before so I wonʼt make a big deal about it here. But what is really needed - and it may already exist in BCP but hasnʼt been promoted is a simple tool: a) start with oneʼs application b) run the app through a tool which recurrsively reads headers c) and creates a list of all headers needed d) extract library names from that list e) add the git repo of those libraries to my apps project. I donʼt seen any way to get around this. And util we can easily do this - boost will not be modular. Robert Ramey _______________________________________________ Unsubscribe & other changes: [3]http://lists.boost.org/mailman/listinfo.cgi/boost References 1. http://Boost.TypeErasure/ 2. http://Boost.Thread/ 3. http://lists.boost.org/mailman/listinfo.cgi/boost

6 5

[doc][geometry][gil][signals] Broken links for Boost 1.70
by Mateusz Loskot 24 Apr '19

24 Apr '19

Hi, On https://www.boost.org/doc/libs/1_70_0/ links to these documentations are broken: https://www.boost.org/doc/libs/1_70_0/libs/geometry/index/doc/html/index.ht… https://www.boost.org/doc/libs/1_70_0/libs/geometry/extensions/doc/html/ind… https://www.boost.org/doc/libs/1_70_0/libs/gil/doc/html/index.html https://www.boost.org/doc/libs/1_70_0/libs/signals/ Could anyone help to identify what is going wrong with those and how to fix it? p.s. I considered posting to boost-docs but that list seems inactive. Best regards, -- Mateusz Loskot, http://mateusz.loskot.net

5 5

CMake, modular Boost, and other stories
by Mike 23 Apr '19

23 Apr '19

I'm very much in favor of separate packages. James is right, that - at the moment - most boost libraries are so tightly coupled that they can't really be treated as separate libraries, but when changing things it is just as important to ask, "Where do you want to go?" as "where you are now". I for one very much hope that boost (at least the libraries that are actively maintained) will become less coupled in the future, such that when I just want to use library X, the rest of boost doesn't come as a dependency with it. It also fits the management structure (or rather the lack thereof) much better. Also, the mechanisms for modular boost distributons already exist through e.g. vcpkg and conan, so I don't see the problem in also providing independent cmake configfiles. Finally: About how many libraries are we talking here anyway? Only the compiled ones get their separate config file - correct? Best Mike

3 2

[GSOC2019][Boost.uBLAS]
by swornim baral 23 Apr '19

23 Apr '19

Hi List, I realize the deadline has passed for seeking a mentor for GSOC, and I apologize for the delay in initiation. I came to know about the program two weeks prior to the deadline, and started working on completing my proposal and competency test prior to finding any mentors. However, I am still very much interested in working on the project of adding a new numerical and multi-linear algebra algorithm to Boost.uBLAS library. I would love to work on the algorithm for calculating Kronecker and Hadamard product with the guidance of a potential mentor. Since I submitted the proposal in a hurry and without any discussion with a mentor, I could not include specific implementation details in my proposal. Below is the link to my final proposal that I submitted for the GSOC 2019. I would really appreciate any feedback on the proposal, competency test and suggested reading materials to get a head start on implementing the algorithm from any interested mentors. https://drive.google.com/open?id=1saUedBfMCOpmVqQfn1a3YcfPtjPYLeDC . Sincerely, Swornim baral

1 0

Re: [boost] CMake, modular Boost, and other stories
by Peter Dimov 23 Apr '19

23 Apr '19

> Mike wrote: > > > Finally: About how many libraries are we talking here anyway? Only the > > compiled ones get their separate config file - correct? > > ~30-40 at last count. Yes, at the moment. More like 50. Some libraries install more than one.

1 0

[variant2] Formal review
by Bjorn Reese 20 Apr '19

20 Apr '19

I. DESIGN --------- The main design decision of Variant2 is the never-empty guarantee, which is relevant when replacing the value of a variant. This is done without overhead if the alternative types are "well-behaving"; if they are not, then Variant2 falls back to using double storage. This is a major difference to: * std::variant which enters an invalid (valuless_by_exception) state. While that may be appropriate for some use cases, there are others where it is not. For instance, the variant may be used for a state machine where the alternative types are function object types to be invoked when events arrive. Having a valueless state is the wrong choice for this use case. * Boost.Variant which uses a temporary heap object instead. Avoiding this internal heap allocation is sufficient grounds for another never-empty variant type. In time critical applications heap allocation not only introduces indeterministic timing, but may also incur priority inversion due to thread synchronization within the heap allocator. Variant2 has several ways to ensure the never-empty guarantee: V1. If all alternative types have non-throwing assignment, the variant cannot fail during re-assignment. V2. If an explicit null state is available, then this is chosen on re-assignment error. This is similar to std::variant, except is is opt-in. V3. If one of the alternative types has a non-throwing default constructor, then this is chosen on re-assignment failure. V4. Otherwise, the variant will remain it its old state (due to double storage) if re-assignment fails. Criterion V3 is possibly surprising because the variant may change to another type not anticipated by the user. This should be opt-in instead (e.g. using a tag.) I am missing a function to get the index of a type to work with the index() function. Something similar to holds_alternative() but that returns the index instead of a boolean. This useful in switch statements, when we want to avoid visitors for some reason: variant<int, float> var = 1; switch (var.index()) { case index_of<int>(var): /* Do int stuff */ break; case index_of<float>(var): /* Do float stuff */ break; } It would also be beneficial for performance reasons to add accessors that have a narrow-contract, similar to relaxed_get() in Boost.Variant. The implementation is already there (detail::unsafe_get.) Better compiler error messages would be desirable. For instance, if we use an illegal visitor that has different return types, then the compiler error does not give us any hint about the different return types. II. IMPLEMENTATION ------------------ The implementation is high quality. Although it does lots of meta-programming, most of the style is simple and easy to follow, especially if you have a rudimentary knowledge of Boost.Mp11. In a few places, like the visit() return type, the meta-programming is more hardcore. variant<T...> defines _get_impl as a public "private" function. This can be made more safe by using detail::unsafe_get instead of _get_impl throughout the code. detail::unsafe_get will continue to call _get_impl, but the former should be made a friend of variant<T...> such that _get_impl can be made private. The other public "private" function, variant<T...>::_real_index(), does not seem to be used and can be removed. The emplace_impl(mp_false, mp_false, ...) contains two distinct cases that depend on a compile time condition. If this is split in two, then the two assert()s can be changed into static_assert()s. The variant_alternative and get functions have a couple of compiler workarounds. The code could be made clearer if the workaround is always used for all compilers. Or does is this related to C++11 constexpr? III. DOCUMENTATION ------------------ The documentation is mainly aimed at experienced users. There is no tutorial for people unfamiliar with variant types, nor is there a design rationale that explains why Variant2 is exists when we already have Boost.Variant and std::variant. The reference documentation is adequate, although a bit terse. IV. MISC -------- Having written my own variant class (that has a maximum size, and any type exceeding the limit is allocated on heap) puts me in a good position to assess Variant2. I was also the review manager of Boost.Mp11 upon which Variant2 is built. I have spent 5-6 hours reading documentation, reviewing the code, and writing small test programs. V. VERDICT ---------- Variant2 fills a hole not covered by Boost.Variant and std::variant. I vote to CONDITIONALLY ACCEPT Variant2 into Boost, provided that the following change be made: 1. The selection of an arbitrary non-throwing type in case of an re-assignment failure (criterion V3 in the DESIGN section above) should be made optional. By default it should use double storage in this case. I would furthermore like to see improvements to the documentation, as well as the index_of/safe_get functionality, but neither is a requirement for acceptance. I have not reviewed expected<T, E...>.

11 29

[boost-ci] Build failures for repositories using boost-ci
by James E. King III 19 Apr '19

19 Apr '19

Recent changes to boost have exposed some shortcomings in the boost-ci infrastructure, specifically the use of CXXFLAGS as a variable to hold b2-specific "cxxflags=" content is now getting passed through to the bootstrap build and going to the compiler, and failing. I'm aware of the issue and I am working on a fix that will be backwards compatible with existing Travis CI scripts that use boost-ci to drive their builds. There may also be changes for Windows, but I haven't looked at that yet. This affects the majority of the CMT repositories as well as date_time, format, uuid, function, move, and signals2, and possibly others. More information to follow... - Jim

1 1

[serialization] clarify in doc on overview that library does not aim for inter-process communication nor for backward compatibility
by rafalcpp＠peerfreedom.org 18 Apr '19

18 Apr '19

Hello, Some developers expect archiving or serializing libraries to be backward compatible, for example if I simply save vector<string> to a JSON, then files written from program with more modern json library will be readable on older one and vice-versa. Boost serialization is instead rather intended to read files created by same program with same boost version (or read it on a more recent, but not on older, version of boost). Could we clarify this with a warning on the Overview page? Perhaps something along this lines: " Warning: please note that the format version may change with boost.archive library versions, an old program may not be able to read a format that was designed after it was build. Explanation: some archive libraries offer long lasting forward and backward compatibility of the archived data. For example if your program saves list of book titles into a JSON file, then if your main program itself does not change you can expect to always be able to load that json file - even when loading it on old version of JSON library while archive was saved on a very new one. This is however *not* the case for this boost archive library (at least for the default archive classes) - here even if your main program stays the same then archives written on one using more recent libboost might be unreadable on ones using older libboost. This also means that communication between users (e.g. over Internet) by exchanging archives created by this library will not work in case if users build the program on their own and build it against different versions of boost. Although, a work around exists in form of including specific version of boost serialization library, and freezing that version, in your program, if you are willing to stick to one version of this module. " Please correct if there are any mistakes (for example if that is possible in some special way).

2 1

[variant2] Review
by Jan Herrmann 17 Apr '19

17 Apr '19

Hello, I looked at the review, at the code and some time ago I tried to implement my own variant. In my opinion in variant2 the individual parts are more worth than the whole. Most implementation details would be of great use for implementing variants with a differnt trade-off. Hence the library ought to be split into several independend parts: * a typing switch * a variadic union * an extendable visitation mechanism * a variant view * related type traits * several implementation of a concrete variant 1. A typing switch The typing switch would be provide a litle bit more than (and can be implementated with help of) mp_with_index. The following pseudo code shows the implementation idea: template<size_t MaxIndex, class Function, class Cast, class Object> constexpr decltype(auto) typing_switch(size_t index, Function fun, Cast cast, Object obj) { return fun(cast(integral_constant<size_t, index>{}, obj)); } The object is a variant and the cast transfroms this special variant to a concrete value (with type). The function is provided by the variant user. The visit function (line 1732) would become something like this: template<class Variant> struct variant_cast { template<class Index> constexpr decltype (auto) operator()(Index, Variant& V) { return unsafe_get<Index::value, V>(); } }; template<class Fun, class Variant> constexpr decltype(auto) visit( F&& f, V1&& v1 ) { return typing_switch<variant_size<V1>>(v.index(), f, variant_cast<V1>, v1); } A simplified deserialisation can be implementated as follows: template<class Types> struct index_to_type { template<class Index> constexpr decltype (auto) operator()(Index, std::nullptr_t) { return mp_identity<mp_at<Types, Index>>{}; } }; template<class Variant, class Archive> Variant deserialize(Archive& ar) { size_t index; ar >> index; Variant f = [&ar](auto type) { decltype(type)::type concrete_element; ar >> concrete_element; return Variant{std::move(concrete_element)}; }; return typing_switch(index, f, index_to_type<Variant>{}, nullptr); } With help of this function an implementation of a visitation function is easy. 2. A varidic union Without doubt a type implementing a varidaic union is a building block for almost every variant implementation. It should be part of the public interface, well tested and with a minimal interface (get<i> and emplace<i>) 3. An extendable visitation mechanism This could be used to implement visitation on multiple different variants. It should use visit on one variant as customization point. 4. A variant view (and a mutable variant view) It could have the following form: template<class... T> class variant_view { void* value; size_t index; public: /* visit ... */ }; It is a little bit like folly/DiscriminatedPtr.h (https://github.com/facebook/folly/blob/master/folly/DiscriminatedPtr.h) but without platform specific code. It can be used for functions which take variants as a const references (or in the mutable form as variants which can change the value but not the actual type). It can be constructed from corresponding variants, values (which are part of T...) and other variant_views. As long as it is constructed from a variant implemented with the variadic union the process can be simple: all members of a union have the same address as the union and the index is equal. For special cases the process can be more complicated. For other variant_views the transformation can be implemented as a runtime lookup in an array and for simple values it is simple. A variant_view can for example be used for functions accepting const references like output and serialisation, or for referencing "polymorphic" objects like nodes in ASTs. 5. Related type traits Meaningfull type traits which can be used in one variant have a good chance be be used in a related one. If one variant want to static assert in case void is one of its types the logic is probably usefull for others. They should be collected. 6. Several implementation of a concrete variant This is the place for variant2. With this building blocks it should be simpler to implement a custom variant. Now lets think of a few use cases: A class contains 3 variants each containing only a few "small" types. Instead of using 3 variants it could pack all indices into one int and construct variant_views as needed. This can reduce the size of the class and saves memory bandwidth. Editing of variants would be of course more complicated. A variant containing a few trivial small types which are used in allmost all cases and one type (T_evil) which is used rarely, is big and can throw in certain situation. The soloution can be a normal variant with the exception of storing T_evil as a smart pointer and dereferences it when visited. A variant with allocator support. In case a user has a solution (or idea) for their variant which lives in shared memory a simple blueprint with generic components like varidic union, a typing switch and useable type traits may be of help. I think requirements for different variant types mentioned in the review could be implemented with some well defined basic building blocks. Beside these thougths I looked at the library. expected.hpp: I tried to compile the following file (containing only one non empty line): #include <boost/variant2/expected.hpp> and it does not compile. It should be a self-contained header. Therefore it should be not part of the library. variant.hpp: Personally (in contrast to other reviewers) I would prefer code more structured in different files. In an ideal world there is for every component a single header file (in clean C++ 17) with minimal dependencies which directes for compiler (and version) specific stuff to a different header. For the future it would result in source code written in newest standard without disturbing influence from ifdefs caused by old buggy compilers. The source has more than 1800 lines with only a few comments, short (1 char) names and very long lines (200 chars and more). For me it is not easy to read (after clang-format it is 2030 lines). Variant alone has more than 450 lines of code. With more shorter files the connection of (and dependencies among) the components would be more visble. I started at the begining of the file and searched for a testcases for monostate and I didn't found one. There are a few testcases inside variant_valueless.cpp which use monostate but none which test it directly. Next variant_size. There are testcases but most deal with variants of type 'variant<> const', 'variant<void>' and the like. Declaring variables of these types results in compile time errors. It is not wrong std::variant gave me the same result, but it is a kind of strange that most tests are implemented in terms of variants wich are ill-formed. On the other hand there are no tests which do not compile. These are two example which should not compile. The template detail::variant_alternative_impl looks like something which copies const, volatile and references. Is there a difference to copy_cv_ref at the end? Could it be used? At the end of the file the comparison is implemented. But operator!= is not implementated in terms of operator== and operator<= and others not in terms of operator<. In case there is a reason for it documentation would be nice. Summarized I think variant2 should NOT be part of boost. I think the focus should be more on reusable components and better documented code. I think most of the code could be very usefull with more descriptive names and test. Regards Jan Herrmann

7 10