Re: [boost] CMake and Boost Build tests

Subject: Re: [boost] CMake and Boost Build tests
From: Florent Castelli (florent.castelli_at_[hidden])
Date: 2017-07-28 01:53:06

On 27/07/2017 23:53, Edward Diener via Boost wrote:
> On 7/27/2017 5:15 PM, Florent Castelli via Boost wrote:
>>
>>> On 27 Jul 2017, at 22:29, Edward Diener via Boost 
>>> <boost_at_[hidden]> wrote:
>>>
>>> On 7/27/2017 12:58 PM, Florent Castelli via Boost wrote:
>>>> On Jul 27, 2017 16:41, "Edward Diener via Boost" 
>>>> <boost_at_[hidden]>
>>>> wrote:
>>>> On 7/27/2017 8:43 AM, Florent Castelli via Boost wrote:
>>>>> On 26/07/2017 20:49, Edward Diener via Boost wrote:
>>>>>
>>>>>> Following John Maddock's appeal for practical solutions related 
>>>>>> to the
>>>>>> move to CMake, I would like to know what the CMake equivalent is 
>>>>>> to the
>>>>>> Boost Build unit test functionality.
>>>>>>
>>>>>> In other words what do I write for CMake in order to do a Boost 
>>>>>> Build
>>>>>> compile, compile-fail, link, link-fail, run, and run-fail unit 
>>>>>> tests ?
>>>>>>
>>>>>
>>>>> In my own Boost-CMake project, I have implemented regular "RUN" in 
>>>>> a way
>>>>> that looks similar to the original Boost Build using functions:
>>>>> https://github.com/Orphis/boost-cmake/blob/master/libs/system.cmake
>>>>>
>>>> What is RUN supposed to be in your link above ?
>>>> An equivalent of run from Boost Build. See
>>>> https://github.com/boostorg/system/blob/develop/test/Jamfile.v2
>>>
>>> Please show the code for RUN in CMake. To ease the transition from 
>>> Boost Build to CMake we really need CMake equivalents to the unit 
>>> testing rules of Boost Build at the very least.
>>
>> Let’s start over. Basically, you end up with those equivalents.
>> Run is compiling, linking an executable and running it, checking the 
>> return code is success. It could translate to:
>>
>> add_executable(test_name test_file.cpp)
>> target_link_libraries(test_name PRIVATE boost_library)
>> add_test(NAME test_name COMMAND test_name)
>>
>> But, if you do that, you need 2 steps to run the test. First you 
>> compile the program (using regular "make test_name" or even “cmake 
>> —build . —target test_name”) and then you run your test through ctest 
>> “ctest . -R test_name” (or you run all the tests without the filter). 
>> The problem there is you have 2 commands, and ctest won’t capture the 
>> compilation or linking errors.
>> To work around that, you can use this instead:
>>
>> add_test(NAME test_name COMMAND cmake —build . —target test_name && 
>> $<TARGET_FILE:test_name>)
>>
>> This will for each test build it with the proper build tool (make, 
>> ninja, msbuild, xcodebuild…) and then run the program. The whole 
>> output will be captured by ctest and can be transfered to any test 
>> dashboard, you won’t have to look for errors in 2 places.
>> The big issue I talk about later if that if you asked ctest to run 
>> several tests in parallel, it will have several “cmake —build . 
>> —target …” in parallel for those tests, which is not supported by 
>> some tools (mainly Ninja).
>>
>
> Understood.
>
>>>
>>>> Other commands should be easy enough to implement as well, but I 
>>>> haven't
>>>>> had the interest in doing that in my project yet (which predates 
>>>>> the SC
>>>>> decision).
>>>>>
>>>>> It is possible to do "compile" by creating a regular static 
>>>>> library target
>>>>> with that file and then invoking "cmake --build . --target 
>>>>> <compile test
>>>>> name>". Compile failures can be done similarly and check they fail.
>>>>> Link and Link-fail can be done the same way using actual binaries 
>>>>> that are
>>>>> built, but not run.
>>>>>
>>>> This is a poor solution.
>>>> Could you please explain why? Using Cmake directly to build the 
>>>> sources
>>>> with the same flags as set in the toolchain is not controversial 
>>>> I'd say.
>>>> It's actually working better than other solutions I've seen trying to
>>>> reproduce the features.
>>>
>>> I said it was a poor solution because all the Boost Build 'compile' 
>>> rule does is to try to compile one or more source files. Success is 
>>> if there are no compilations errors, while failure is if there is at 
>>> least one compilation error. Why should one have to create an actual 
>>> library to do this, especially as the source(s) may contain a main() 
>>> function, and in fact usually does ?
>>>
>>> I think the right solution, as suggested to me by someone else, is 
>>> to create a CMake OBJECT library, which as I understand it is not 
>>> really a library but just OBJECT files, and then run a test based on 
>>> that. But of course I do not know if CMake supports that.
>>
>> The thing is, static libraries are guaranteed to work and fairly 
>> standard, object libraries are a bit finicky and may not work 
>> depending on the generator. Using one or the other is merely an 
>> optimisation though, static libraries are just an archive of object 
>> files after all. Similar to the code above, you could implement them 
>> with this code:
>>
>> add_library(test_name STATIC (or OBJECT) test_file.cpp)
>> target_link_libraries(test_name PRIVATE boost_library)
>> add_test(NAME test_name COMMAND cmake —build . —target test_name)
>>
>> In this code, ctest will just build it and capture any compilation 
>> error. Easy enough. It doesn’t matter if there are functions called 
>> main or not since it’s not linked, the compiler won’t care for the 
>> most part about the name of the functions (well, main has some 
>> special semantics, but I doubt any test would rely on that).
>
> But now you are filling up an end-user's build output with a static 
> library for each 'compile' tested. Ughhh ! That seems so silly to me, 
> where just adding object files on success seems normal.

I understand that, but some IDEs may have issues with those. I remember 
having issues with Xcode a while ago. Requires more testing I would say.

>
>>
>> For completeness, here’s a possible implementation of “LINK”. I 
>> understand this one links the source files into a binary, but doesn’t 
>> run it. This is simple CMake code as well:
>>
>> add_executable(test_name test_file.cpp)
>> add_test(NAME test_name COMMAND cmake —build . —target test_name)
>>
>> In this case, CMake will build and link the files from the ctest 
>> invocation.
>>
>> To support the failing variants for all the above, you need this
>> set_test_properties(test_name PROPERTIES WILL_FAIL TRUE).
>> Note that if you expect a test to fail running, but still succeed 
>> linking, that’s not directly supported by the code above and some 
>> changes need to be made.
>>
>> Overall, that’s a lot of boilerplate for each test, which is why you 
>> need some convenience functions to make the build files clean and 
>> simple, with a clear intent.
>> In complex test cases, it will certainly be possible to have people 
>> using CMake commands directly, but that should be rare.
>>
>
> Understood. Thanks !
>
>
> _______________________________________________
> Unsubscribe & other changes: 
> http://lists.boost.org/mailman/listinfo.cgi/boost