Re: [boost] CMake - one more time

Subject: Re: [boost] CMake - one more time
From: Raffi Enficiaud (raffi.enficiaud_at_[hidden])
Date: 2016-04-22 21:39:33

Le 23/04/16 à 02:30, Paul Fultz II a écrit :
>
>> On Apr 22, 2016, at 4:56 PM, Raffi Enficiaud <raffi.enficiaud_at_[hidden]> wrote:
>>
>> Le 22/04/16 à 19:42, Paul Fultz II a écrit :
>>>
>>>> On Apr 20, 2016, at 4:39 PM, Raffi Enficiaud <raffi.enficiaud_at_[hidden]> wrote:
>>>> <snip>
>>>> I do not like the current state of b2 for many reasons (even though I think it could be really a good build system), but CMake is not covering many features that are currently required by the boost superproject. Until the point where we can consistently build the lib (including the possibly many flavor of the same library - STATIC/SHARED at the same time, etc), run the tests, and generate the documentation (including the possibility to have the boostdoc/quickbook/doxygen toolchain), I do not see any *good* reason to move to cmake.
>>>
>>> Cmake can do all that you listed there. In addition, I found its easy to generate targets at configure time. So the Fit library actually adds additional tests to test examples and header files. Plus, I can create configuration headers based on whether a something compiles or runs. I believe BB can do the same using virtual targets or something, but I haven’t clearly figured it out.
>>>
>> Certainly, CMake can do everything with the appropriate effort. But so far, although I am a CMake user, I do not know how to do this:
>>
>> - having the same target name with different target properties: like
>>  set(PROJECT_SRC ....)
>>  add_library(myproject SHARED ${PROJECT_SRC})
>>  add_library(myproject STATIC ${PROJECT_SRC})
>>
>>  how do you do that? how do you refer to the appropriate variant?
>
> In cmake, you can create object libraries:
>
> https://cmake.org/cmake/help/v3.5/command/add_library.html#object-libraries
>
> This help avoid the double compile, so you can write this:
>
> add_library(objlib OBJECT ${SOURCES})
> # shared libraries need PIC
> set_property(TARGET objlib PROPERTY POSITION_INDEPENDENT_CODE 1)
> add_library(MyLib_shared SHARED $<TARGET_OBJECTS:objlib>)
> add_library(MyLib_static STATIC $<TARGET_OBJECTS:objlib>)
>
> Then to set each one to the same name you can use the OUTPUT_NAME property:
>
> set_target_properties(MyLib_shared PROPERTIES OUTPUT_NAME MyLib)
> set_target_properties(MyLib_static PROPERTIES OUTPUT_NAME MyLib)

Exactly, so you artificially make CMake think that 2 different targets 
should end up with the same name on the filesystem. It does not work for 
instance on Win because the import .lib of the shared get overwritten by 
the static. This is not exactly a solution, but rather a hack (or 
workaround). We can of course iterate further (set the output folder per 
type, etc).

>> - having a set of dependencies that is not driven by a high level CMakeLists.txt. You advocate the solution of packaging, but this does not target all platforms,
>
> How does this not target all platforms?

Do I have a centralized (or virtualized like inside vagga/docker or 
virtualenv) and official packet manager on Win32 or OSX? I know tools 
exist (brew, chocolatey, etc). What about the other platforms (Android)? 
What about cross compilation?

>
>> and just translates the same problem to another layer to my opinion. As a developer, in order to work on a library X that depends on Y, you should install Y, and this information should appear in X (so this is an implicit knowledge). What this process does is that it put X and Y in some same level of knowledge: a flatten set of packages. This is done by BJam already the same, but at compilation/build step, and without the burden of the extra management of packages (update upstream Y for instance, when Y can be a set of many packages, and obviously in a confined, repeatable and isolated development environment). But maybe you think of something else.
>
> I don’t follow this at all. For example, when I want to build the hmr library here: https://github.com/pfultz2/hmr
>
> All I have to do after cloning it is: `cget build`, then it will go and grab the dependencies because they have been listed in the requirements.txt file.

Then I am dependent on another tool, cget, maintained by ... you :) Also 
from the previous thread, if my project has not the "standard" cget 
layout, then cget will not work (yet?). I also need another file, 
"requirements" that I need to maintain externally to the build system. I 
do that often for my python packages, and it is "easy" but difficult to 
stabilize sometimes, especially in complex dependency graph (and their 
can be conflicting versions, etc). I can see good things in cget, I can 
also see weak points.

What I am saying is that you delegate the complexity to another layer, 
call it cget or Catkin, or pip. And developers should do also the 
packaging, and this is not easy (and needs a whole infrastructure to 
make it right, like PPAs or pypi).

BTW, is cget able to work offline?

>>
>> To me this is a highly non trivial task to do with CMake, and ends up in half backed solutions like ROS/Catkin (http://wiki.ros.org/catkin/conceptual_overview), which is really not CMake and is just making things harder for everyone.
>
> Cmake already handles the packaging and finding dependencies, cget just provides the mechanism to retrieve the packages using the standard cmake process. This why you can use it to install zlib or even blas, as it doesn’t require an extra dependency management system.

Well, I really cannot tell for cget. CMake finds things that are 
installed in expected locations for instance, otherwise the FIND_PATHS 
should be indicated (and propagated to the dependency graph). What if 
for instance, it needs an updated/downgraded version of the upstream? 
How cget does manage that?

Is there an equivalent to virtualenv? Right now for boost, I clone the 
superproject, and the artifacts and dependencies are confined withing 
this clone (up to doxygen, docbook etc).

>
>>
>> - I can continue... such as targets subset selection. It is doable with CMake with, "I think" some umbrella projects, but again this is hard to maintain and requires a high level orchestration. Only for the tests for instance: suppose I do not want to compile them in my first step, and then I change my mind, I want to run a subset of them. What I also want is not wasting my time in waiting for a billion of files to compile, I just want the minimal compilation. So it comes to my mind that EXCLUDE_FROM_ALL might be used, but when I run ctest -R something*, I get an error... Maybe you know a good way of doing that in cmake?
>
> I usually add the tests using this(I believe Boost.Hana does the same):
>
> add_custom_target(check COMMAND ${CMAKE_CTEST_COMMAND} -VV -C ${CMAKE_CFG_INTDIR})
>
> function(add_test_executable TEST_NAME)
>     add_executable (${TEST_NAME} EXCLUDE_FROM_ALL ${ARGN})
>     if(WIN32)
>         add_test(NAME ${TEST_NAME} WORKING_DIRECTORY ${LIBRARY_OUTPUT_PATH} COMMAND ${TEST_NAME}${CMAKE_EXECUTABLE_SUFFIX})
>     else()
>         add_test(NAME ${TEST_NAME} COMMAND ${TEST_NAME})
>     endif()
>     add_dependencies(check ${TEST_NAME})
>     set_tests_properties(${TEST_NAME} PROPERTIES FAIL_REGULAR_EXPRESSION "FAILED")
> endfunction(add_test_executable)
>
> Then when I want to build the library I just run `cmake —build .` and then when I want to run the test, I can run `cmake —build . —target check`. Now if I want to run just one of the tests I can do `cmake —build . —target test_name && ./test_name` just as easy. I have not ever had the need to run subset of tests, this is usually the case when there is nested projects, but is easily avoided when the project is separated into separate components.

You are strengthening my point, you write an umbrella target for your 
purpose. My example with the tests was a trap: if you run "cmake —build 
. —target check" you end up building "all" the tests. To have a finer 
granularity, you should write "add_test_executable_PROJECTX" etc. BJam 
knows how to do that, also with a eg. STATIC version of some upstream 
library, defined at the point it is consumed (and not at the point it is 
declared/defined), and built only if needed, without the need to do some 
mumbo/jumbo with object files.

What I am saying is that it is indeed possible, I also know solutions,
but this is not native to cmake.

>>> Finally, for boost, it could provide some high-level cmake functions so all of these things can happen consistently across libraries.
>>
>> Sure. Or ... BJam should be given some more care and visibility, like a GSoC (bis) track?
>
> But its not entirely technology that is missing, its the community that is missing, and I don’t think a GSoC will help create a large community for boost build.

That is true. I see it as an chicken and egg problem also, and we have 
to start somewhere.

Where Bjam will always loose is the ability to generate IDE 
environments, natively, and this is a major reason why cmake will have a 
more lively community. I believe that a BJam to cmake is possible, but 
even in that case, Bjam will live in the shadow of cmake.

I was proposing GSoC for, eg. start thinking about the syntax in bjam ...

Raffi