Re: [boost] Boost CMake support - Request for Comment

Subject: Re: [boost] Boost CMake support - Request for Comment
From: Edward Diener (eldiener_at_[hidden])
Date: 2018-10-09 23:42:38

On 10/9/2018 7:09 PM, mike via Boost wrote:
>> -----Original Message-----
>> From: Boost <boost-bounces_at_[hidden]> On Behalf Of Edward Diener via
>> Boost
>> Sent: Monday, October 8, 2018 10:22 PM
>>
>> On 10/8/2018 6:15 AM, Mike Dev via Boost wrote:
>>>> -----Original Message-----
>>>> From: Boost <boost-bounces_at_[hidden]> On Behalf Of Edward Diener
>>>> via Boost
>>>> Sent: Monday, October 1, 2018 8:00 PM
>>>>
>>>> On 10/1/2018 1:07 PM, Robert Ramey via Boost wrote:
>>>>> Background
>>>>> ==========
>>>>>
>>>>> My personal knowledge of build systems is somewhat limited.
>>>>>
>>>>> a) I don't really understand CMake all that well.  I have made
>>>>> CMakeLists.txt files for both the serialization library and the safe
>>>>> numerics libraries as an aid to build and maintain IDE and make files
>>>>> used in running tests and posting results to CDash.
>>>>>
>>>>> b) I don't really understand our current system, Boost Build, very
>>>>> well
>>>>> either.  I've made jam files for building and running tests for the
>>>>> serialization and safe numerics libraries.
>>>>>
>>>>> c) I recently reviewed C++Now 2017: Daniel Pfeifer's excellent video
>>>>> from C++Now 2017  “Effective CMake" -
>>>>> https://www.youtube.com/watch?v=bsXLMQ6WgIk . I recommend it highly.
>>>>>
>>>>> Despite my lack of knowledge in this area, or perhaps because of it,
>>>>> I've volunteered to direct the effort to evolve Boost so that users of
>>>>> Boost can avail themselves of the benefits of CMake.  This effort will
>>>>> be organized as follows:
>>>>>
>>>>> a) A discussion on the boost developers mailing list with the goal of
>>>>> reaching a consensus as to what benefits CMake can and should provide
>>>>> to
>>>>> users and developers of Boost.  This effort will commence with the
>>>>> posting of this notice.  I hope that this discussion can be more or
>>>>> less
>>>>> resolved within 30 days.
>>>>>
>>>>> b) I will then synthesize from the above discussion a call for
>>>>> proposals
>>>>> which lists the requirements and requirements.
>>>>>
>>>>> c) Those proposing CMake for boost will have approximately 90 days to
>>>>> prepare their proposals.  Their proposal are expected to look similar
>>>>> to
>>>>> a boost library proposal.  That is they are expected to have some
>>>>> (CMake) code, along with tests, documentation including tutorials and
>>>>> reference.  This is only an expectation.  It's conceivable that such a
>>>>> proposal might only contain conventions and and samples of what boost
>>>>> library should contain to fulfill the requirements. As with boost
>>>>> libraries, submissions are not required to totally complete.  But they
>>>>> have to be sufficiently complete to convince reviewers the the
>>>>> submission fulfills the stated requirements and can be finished in
>>>>> good
>>>>> time.
>>>>>
>>>>> d) After the 90 day period, submissions will be reviewed
>>>>> simultaneously.
>>>>>     Note that this is at a variance from normal boost procedure of
>>>>> reviewing submissions one at a time.  Since we expect to receive a
>>>>> number of submissions and only one can be accepted, the normal boost
>>>>> protocol can't really function.  Target date for this review is 1
>>>>> February 2019.
>>>>>
>>>>> e) After the review period the review manager will prepare a report
>>>>> which includes the decision of which submission will be accepted into
>>>>> the official boost distribution.  Currently, I, Robert Ramey, expect
>>>>> to
>>>>> be review manager.  However, it's possible that this by the time the
>>>>> review is undertaken, this coveted position could be assigned to some
>>>>> more worthy candidate.
>>>>>
>>>>> Scope, Requirements and Features
>>>>> ================================
>>>>>
>>>>> We presume that submissions will fulfill the applicable requirements
>>>>> of
>>>>> any boost submission.  In addition we would like to see the following
>>>>> addressed.
>>>>>
>>>>> Library Build
>>>>> -------------
>>>>>          “Effective CMake" -
>>>>> https://www.youtube.com/watch?v=bsXLMQ6WgIk
>>>>> Recommends that all libraries - including header only libraries have a
>>>>> CMakeList.txt file.  What should this include for Boost Libraries?
>>>>>
>>>>>
>>>>> Library Test
>>>>> ------------
>>>>>        Should facilities for "testing" be only done by developers? or
>>>>> should users who acquire library packages also be able to test
>>>>> libraries.
>>>>>        Should CMake testing results posting be used - CDASH?
>>>>>        Should we just skip the issue of Library Testing and continue to
>>>>> depend on Boost Build.
>>>>>
>>>>> Library Packaging
>>>>> -----------------
>>>>>        Is the library packaging facility provide by CMake - CPACK -
>>>>> useful
>>>>> to boost.  Should boost libraries be updated to support it?
>>>>>
>>>>> Dependencies
>>>>> ------------
>>>>>        If the above is implemented, can we depend upon CMake to handle
>>>>> library dependencies?  Or does some special functionality
>>>>>        Do circular dependencies constitute a problem?
>>>>>
>>>>> Modularity
>>>>> ----------
>>>>>        Currently boost is organized as a tightly integrated group of
>>>>> libraries.  This organization manifests itself in a number of ways.
>>>>>            We have a Boost "super project" in github.  Boost libraries
>>>>> are
>>>>> "sub projects" of this super project.  This known structure is
>>>>> exploited
>>>>> by and depended upon by boost tools.  Should CMake support continue
>>>>> this
>>>>> policy/design.
>>>>>            We distribute boost as "super project".  Should we continue
>>>>> to
>>>>> do this or distribute libraries on a library by library basis.  Does
>>>>> CMake/CPACK etc. make this "simple"?
>>>>>
>>>>> Documentation/testing
>>>>> ---------------------
>>>>>        What support should be require for uses of Boost CMake
>>>>> implementation. Documentation, examples, templates for helping library
>>>>> developers.  How should the CMake design/implementation be tested?
>>>>>
>>>>> Other considerations
>>>>> --------------------
>>>>> add your requirement/features here.
>>>>>
>>>>>
>>>>> Useful Resources
>>>>> ===============
>>>>>        “Effective CMake" - https://www.youtube.com/watch?v=bsXLMQ6WgIk
>>>>>        https://gist.github.com/mbinna/c61dbb39bca0e4fb7d1f73b0d66a4fd1
>>>>>        https://github.com/purpleKarrot/Boost.CMake
>>>>>        https://www.youtube.com/watch?v=eC9-iRN2b04
>>>>>
>>>>> Please try to keep this discussion on this list.
>>>>>
>>>>> I know that there has been a lot going on on slack. But having it in
>>>>> two
>>>>> places makes it harder for me to review and summarize.  Also slack
>>>>> doesn't conveniently maintain comment hierarchy.  So please try to
>>>>> keep
>>>>> this discussion on this list.
>>>>
>>>> I strongly suggest that the goal of using CMake with Boost begin with
>>>> making Boost libraries, whether header-only or built, available to
>>>> CMake
>>>> projects. This also means that we have a way of testing whether or not
>>>> such a solution actually works for any given Boost library. My own
>>>> preference for testing whether or not such a solution would work or not
>>>> would be to use our current b2/Boost Build test jamfiles, but if this
>>>> is
>>>> too difficult to do we still need some way to test the solution for
>>>> each
>>>> library so each library maintainer can see whether or not the solution
>>>> we choose works for his library.
>>>>
>>>> This should be our first goal and all other goals should be tabled
>>>> until
>>>> and when we can have a solution where this works flawlessly and there
>>>> is
>>>> a general agreement that whatever proposal is chosen to implement this
>>>> is best.
>>>>
>>>> Only when the above is fully accomplished should Boost then look into
>>>> the goal of using CMake internally, perhaps for test, docs, building
>>>> libraries, or what-not. We need to work this way because every solution
>>>> is bound to end with comments that it does X but does not do Y or it
>>>> does X and Y but does not do Z, and we all know where these endless
>>>> arguments end: that we end up doing nothing because we start by trying
>>>> to do too much and we can never reach agreement.
>>>
>>> I agree with you in so far, that the focus should be on the interface
>>> (how to consume boost libraries) and less on the implementation details,
>>> but I see two practical reasons, why it is beneficial to immediately
>>> use cmake (at least) for building too:
>>>
>>> 1) Creating a CMake target for a separately compiled binary that can be
>>>      consumed by the user is about the same effort as using cmake to
>>> build
>>>      the library (if necessary) in which case you get the interface
>>>      information for free (switching to cmake for unit-tests and docs is
>>> a
>>>      different topic).
>>>
>>> 2) "making Boost libraries [...] available to CMake projects" for me
>>> also
>>>      means I have an easy way to compile those libraries with specific
>>>      compiler flags that match my project settings - otherwise, what is
>>> the
>>>      advantage compared to the current mechanism that is already build
>>> into
>>>      cmake?
>>>      Ensuring consistent build flags is relatively easy if all libraries
>>> use
>>>      the same build system (in this case cmake) but can become quite
>>> complex
>>>      if you want to implement an automatic translation from cmake to b2
>>> or
>>>      vice versa.
>>>
>>> So yes, while a library (build-) interface and consummation should be
>>> the
>>> focus of the review, I'd expect that at least building the libraries
>>> with
>>> cmake will be the simplest way to implement that interface.
>>>
>>> Just to be completely clear: Just because the "reference implementation"
>>> of a cmake interface is probably a cmake-based build, that doesn't mean
>>> everyone has to actually adopt that reference implementation as long as
>>> the interface is consistent.
>>
>> Boost Build has a feature which allows the programmer to test at build
>> time what C++ features are supported for a particular compiler
>> implementation and compile parameters, and change the build internally
>> on-the-fly accordingly. I do not know if any Boost non-header only
>> library uses this feature but if one does it will be difficult
>> translating this to CMake terms, if such a similar feature exists in
>> CMake.
> 
> As Richard pointed out, such features exist in cmake too and have been in
> common use for years. E.g. It has a built-in mechanism that lets you
> directly query for most of the post c++03 compiler features:
> (https://cmake.org/cmake/help/v3.13/prop_gbl/CMAKE_CXX_KNOWN_FEATURES.html)
> and as a more general feature, you can always check if a particular source
> file using some feature or type compiles or not:
> (https://cmake.org/cmake/help/v3.13/module/CheckCXXSourceCompiles.html)
> 
> Could you maybe point me to some examples where this feature is used so one
> could estimate how difficult a conversion is?

The feature is essentially programmed by config and predef to provide 
build-time testing of compiler support and testing of such things like 
the OS versions and compiler versions etc. See 
https://www.boost.org/doc/libs/1_68_0/libs/config/doc/html/boost_config/build_config.html 
for the explanation in config and 
https://www.boost.org/doc/libs/1_68_0/doc/html/predef/check_utilities.html 
for checking predef values at build time. The basics of the feature are 
offered by Boost Build as explained at 
https://boostorg.github.io/build/manual/develop/index.html#bbv2.reference.rules 
in reference to the check-target-builds rule. Plenty of libraries use 
the feature during testing and I imagine that some of them may use the 
feature during building. Asking individual libraries to convert their 
tests or build to use some CMake equivalent, for every possible setting 
in config and predef which they use, is not realistic. I understand that 
CMake may have an equivalent technique, but who is going to program that 
technique to offer all the equivalents which config and predef currently 
offer ?

> 
>> It is for such a reason that as soon as you start talking about using
>> CMake for anything else but identifying Boost libraries for the purposes
>> of end-user use of Boost libraries in their own CMake scripts,
> 
> Just consuming precompiled/pre-installed boost libraries in a cmake script
> is already possible thanks to the work of cmake contributors and the people
> providing boost packages for vcpkg, conan and other package management
> systems (some of those contributors are probably also boost library
> maintainers?).
> Those solutions have to keep playing catch up with the latest boost
> release if something significant changes (like name-mangling or a new
> library), but by and large they work quite well. Imho an official boost
>   solution should provide a bit more functionality to justify the effort.
> 
>> that you
>> may run into much more work than you can imagine. You just can't tell
>> individual library developers to translate all their own use of Boost
>> Build into CMake, as if this will be simple for them to do, and they
>> will just do it with little time spent and as if it will be no work for
>> them.
> 
> Let's get one thing straight: Switching to cmake will require work, no
> one is denying that and it will not happen during the course of a single
> release. And if library developers are not willing to invest some time to
> do this, we can forget about it all together. However, it is my impression
> that many developers are willing to make that effort. All that is lacking
> is a clear decision of what the (semi-) final solution should look like to
> make sure the effort isn't wasted.
> 
> Now as I said. I'm not convinced, that only providing interface information
> is actually less work than building boost with cmake and letting it generate
> the cmake information automatically. That probably depends to a large degree
> if those cmake files can be auto generated by b2. However, just as there
> seem to be more complex build scenarios, there are probably scenarios where
> the auto-generated cmake-config files are not enough.
> 
>> Boost Build, for better or worse, has some extensive features
>> which have no easy correspondence to CMake.
> 
> Such as? If plain cmake is lacking any important features that are needed
> by boost, then a replacement is exactly what a boost cmake library should
> provide.
> 
>> Unless you or someone else
>> is willing to develop a bridge between Boost Build and CMake for
>> features of Boost Build which libraries rely on, then individual Boost
>> libraries which use such Boost Build features are going to find it very
>> difficult to transition to CMake, possibly even for something as
>> seemingly innocuous as building the library itself using a CMake script.
> 
> Let's not forget, that there are already solutions out there that build
> (almost?) all of boost with cmake. Here is one:
> https://github.com/boost-cmake/boost
> 
> Now instead of adopting and tweaking one of those existing solutions during
> review, you are imho asking for the development of yet another solution. I'm
> not sure if that is actually an easier path forward.
> That doesn't mean that an individual library author can't decide he wants
> to stick to b2 internally and just provide a cmake interface file, but we
> for the time being and
> 
>>
>> This is why I have urged the move to CMake by Boost to proceed slowly by
>> baby steps, rather than to try doing things that may take a huge initial
>> effort to get done, and therefore may end up never accomplishing anything.
> 
> That is kind of what I'm trying to achieve with my individual mini PRs. They
> are in no way a replacement for the existing boost build system, as they
> only cover a very narrow use case, but even those also build the boost libraries.
> Again, I don't think building is the issue (as I said, it has been done already)
> and if someone really encounters a problem, I'm sure people on the ML and
> elsewhere are willing to help. IMHO it will be far more difficult to agree
> all the bike shedding topics related to how exactly boost libraries should be
> made available to the consumer.
> 
> Also, proceeding by Baby steps means that boost will stay in limbo between
> cmake and boost build for quite some time, where library maintainers will
> have to maintain two systems and more or less constantly upgrading their
> CMake files. It doesn't have to happen all in one big step (I agree that that
> is not realistic), but considering that consumers can already use boost
> libraries in their cmake projects (albeit with a few pain points here and there)
> and that there are already proof of concepts for more fully fledged solutions
> out there, I think the initial goal should be a tad more ambitious.
> 
> 
> Mike