Re: [boost] CMake - one more time

Subject: Re: [boost] CMake - one more time
From: Paul Fultz II (pfultz2_at_[hidden])
Date: 2016-04-23 00:34:00

> On Apr 22, 2016, at 8:39 PM, Raffi Enficiaud <raffi.enficiaud_at_[hidden]> wrote:
> 
> Le 23/04/16 à 02:30, Paul Fultz II a écrit :
>> 
>>> On Apr 22, 2016, at 4:56 PM, Raffi Enficiaud <raffi.enficiaud_at_[hidden]> wrote:
>>> 
>>> Le 22/04/16 à 19:42, Paul Fultz II a écrit :
>>>> 
>>>>> On Apr 20, 2016, at 4:39 PM, Raffi Enficiaud <raffi.enficiaud_at_[hidden]> wrote:
>>>>> <snip>
>>>>> I do not like the current state of b2 for many reasons (even though I think it could be really a good build system), but CMake is not covering many features that are currently required by the boost superproject. Until the point where we can consistently build the lib (including the possibly many flavor of the same library - STATIC/SHARED at the same time, etc), run the tests, and generate the documentation (including the possibility to have the boostdoc/quickbook/doxygen toolchain), I do not see any *good* reason to move to cmake.
>>>> 
>>>> Cmake can do all that you listed there. In addition, I found its easy to generate targets at configure time. So the Fit library actually adds additional tests to test examples and header files. Plus, I can create configuration headers based on whether a something compiles or runs. I believe BB can do the same using virtual targets or something, but I haven’t clearly figured it out.
>>>> 
>>> Certainly, CMake can do everything with the appropriate effort. But so far, although I am a CMake user, I do not know how to do this:
>>> 
>>> - having the same target name with different target properties: like
>>> set(PROJECT_SRC ....)
>>> add_library(myproject SHARED ${PROJECT_SRC})
>>> add_library(myproject STATIC ${PROJECT_SRC})
>>> 
>>> how do you do that? how do you refer to the appropriate variant?
>> 
>> In cmake, you can create object libraries:
>> 
>> https://cmake.org/cmake/help/v3.5/command/add_library.html#object-libraries
>> 
>> This help avoid the double compile, so you can write this:
>> 
>> add_library(objlib OBJECT ${SOURCES})
>> # shared libraries need PIC
>> set_property(TARGET objlib PROPERTY POSITION_INDEPENDENT_CODE 1)
>> add_library(MyLib_shared SHARED $<TARGET_OBJECTS:objlib>)
>> add_library(MyLib_static STATIC $<TARGET_OBJECTS:objlib>)
>> 
>> Then to set each one to the same name you can use the OUTPUT_NAME property:
>> 
>> set_target_properties(MyLib_shared PROPERTIES OUTPUT_NAME MyLib)
>> set_target_properties(MyLib_static PROPERTIES OUTPUT_NAME MyLib)
> 
> Exactly, so you artificially make CMake think that 2 different targets should end up with the same name on the filesystem. It does not work for instance on Win because the import .lib of the shared get overwritten by the static. This is not exactly a solution, but rather a hack (or workaround). 

Its neither, this is optimization because the user could just build the library twice, once for shared and another for static. Of course, this type of optimization mainly affects system maintainers and so everyday users of cmake don’t see this as a big problem.

> We can of course iterate further (set the output folder per type, etc).
> 
>>> - having a set of dependencies that is not driven by a high level CMakeLists.txt. You advocate the solution of packaging, but this does not target all platforms,
>> 
>> How does this not target all platforms?
> 
> Do I have a centralized (or virtualized like inside vagga/docker or virtualenv) and official packet manager on Win32 or OSX? I know tools exist (brew, chocolatey, etc). What about the other platforms (Android)? What about cross compilation?

There is bpm, cget, conan, and hunter to name a few that is cross platform and targets all platforms.

> 
>> 
>>> and just translates the same problem to another layer to my opinion. As a developer, in order to work on a library X that depends on Y, you should install Y, and this information should appear in X (so this is an implicit knowledge). What this process does is that it put X and Y in some same level of knowledge: a flatten set of packages. This is done by BJam already the same, but at compilation/build step, and without the burden of the extra management of packages (update upstream Y for instance, when Y can be a set of many packages, and obviously in a confined, repeatable and isolated development environment). But maybe you think of something else.
>> 
>> I don’t follow this at all. For example, when I want to build the hmr library here: https://github.com/pfultz2/hmr
>> 
>> All I have to do after cloning it is: `cget build`, then it will go and grab the dependencies because they have been listed in the requirements.txt file.
> 
> Then I am dependent on another tool, cget, maintained by ... you :) Also from the previous thread, if my project has not the "standard" cget layout, then cget will not work (yet?).

There is no layout requirements. It just requires a CMakeLists.txt at the top level(which all cmake-supported libraries have), but the library can be organized however.

> I also need another file, "requirements" that I need to maintain externally to the build system.

But building and installing the dependencies is external to the build system anyways. The requirements.txt just lets you automate this process.

> I do that often for my python packages, and it is "easy" but difficult to stabilize sometimes, especially in complex dependency graph (and their can be conflicting versions, etc). I can see good things in cget, I can also see weak points.

Currently cget doesn’t handle versions. I plan to support channels in the future which can support versions and resolve dependencies using a SAT solver(which pip does not do).

> 
> What I am saying is that you delegate the complexity to another layer, call it cget or Catkin, or pip. And developers should do also the packaging, and this is not easy (and needs a whole infrastructure to make it right, like PPAs or pypi).

The complexity is there, which I hope tools like bpm or cget can help with. However, resolving the dependencies by putting everything in a superproject is more of a hack and doesn’t scale. 

> 
> BTW, is cget able to work offline?

Yes.

> 
>>> 
>>> To me this is a highly non trivial task to do with CMake, and ends up in half backed solutions like ROS/Catkin (http://wiki.ros.org/catkin/conceptual_overview), which is really not CMake and is just making things harder for everyone.
>> 
>> Cmake already handles the packaging and finding dependencies, cget just provides the mechanism to retrieve the packages using the standard cmake process. This why you can use it to install zlib or even blas, as it doesn’t require an extra dependency management system.
> 
> Well, I really cannot tell for cget. CMake finds things that are installed in expected locations for instance, otherwise the FIND_PATHS should be indicated (and propagated to the dependency graph).

It sets the CMAKE_PREFIX_PATH(and a few other variables), which cmake uses to find libraries.

> What if for instance, it needs an updated/downgraded version of the upstream? How cget does manage that?

`cget -U` will replace the current version.

> 
> Is there an equivalent to virtualenv? Right now for boost, I clone the superproject, and the artifacts and dependencies are confined withing this clone (up to doxygen, docbook etc).

By default it installs everything in the local directory `cget`, but this can be changed by using the `—prefix` flag or setting the `CGET_PREFIX` environment variable.

> 
>> 
>>> 
>>> - I can continue... such as targets subset selection. It is doable with CMake with, "I think" some umbrella projects, but again this is hard to maintain and requires a high level orchestration. Only for the tests for instance: suppose I do not want to compile them in my first step, and then I change my mind, I want to run a subset of them. What I also want is not wasting my time in waiting for a billion of files to compile, I just want the minimal compilation. So it comes to my mind that EXCLUDE_FROM_ALL might be used, but when I run ctest -R something*, I get an error... Maybe you know a good way of doing that in cmake?
>> 
>> I usually add the tests using this(I believe Boost.Hana does the same):
>> 
>> add_custom_target(check COMMAND ${CMAKE_CTEST_COMMAND} -VV -C ${CMAKE_CFG_INTDIR})
>> 
>> function(add_test_executable TEST_NAME)
>>    add_executable (${TEST_NAME} EXCLUDE_FROM_ALL ${ARGN})
>>    if(WIN32)
>>        add_test(NAME ${TEST_NAME} WORKING_DIRECTORY ${LIBRARY_OUTPUT_PATH} COMMAND ${TEST_NAME}${CMAKE_EXECUTABLE_SUFFIX})
>>    else()
>>        add_test(NAME ${TEST_NAME} COMMAND ${TEST_NAME})
>>    endif()
>>    add_dependencies(check ${TEST_NAME})
>>    set_tests_properties(${TEST_NAME} PROPERTIES FAIL_REGULAR_EXPRESSION "FAILED")
>> endfunction(add_test_executable)
>> 
>> Then when I want to build the library I just run `cmake —build .` and then when I want to run the test, I can run `cmake —build . —target check`. Now if I want to run just one of the tests I can do `cmake —build . —target test_name && ./test_name` just as easy. I have not ever had the need to run subset of tests, this is usually the case when there is nested projects, but is easily avoided when the project is separated into separate components.
> 
> You are strengthening my point, you write an umbrella target for your purpose. My example with the tests was a trap: if you run "cmake —build . —target check" you end up building "all" the tests. To have a finer granularity, you should write "add_test_executable_PROJECTX" etc. BJam knows how to do that, also with a eg. STATIC version of some upstream library, defined at the point it is consumed (and not at the point it is declared/defined), and built only if needed, without the need to do some mumbo/jumbo with object files.

I don’t know see how that is something that cmake doesn’t do either.

> 
> What I am saying is that it is indeed possible, I also know solutions,
> but this is not native to cmake.

Yes, its possible, and a module would help make it possible in a simpler way, although, I don’t know how common it is to group tests. In general, I usually just focus on one test or all the tests.

> 
>>>> Finally, for boost, it could provide some high-level cmake functions so all of these things can happen consistently across libraries.
>>> 
>>> Sure. Or ... BJam should be given some more care and visibility, like a GSoC (bis) track?
>> 
>> But its not entirely technology that is missing, its the community that is missing, and I don’t think a GSoC will help create a large community for boost build.
> 
> That is true. I see it as an chicken and egg problem also, and we have to start somewhere.
> 
> Where Bjam will always loose is the ability to generate IDE environments, natively, and this is a major reason why cmake will have a more lively community. I believe that a BJam to cmake is possible, but even in that case, Bjam will live in the shadow of cmake.

Yep, and instead of competing with cmake, boost could collaborate with cmake and would have a much larger impact.

> 
> I was proposing GSoC for, eg. start thinking about the syntax in bjam …

Yes the bjam syntax is the worst part.

> 
> Raffi
> 
> 
> _______________________________________________
> Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost