Boost logo

Boost :

Subject: Re: [boost] CMake and Boost Build tests
From: Niall Douglas (s_sourceforge_at_[hidden])
Date: 2017-07-27 12:34:36


>> Actually cmake's support for fail tests is richer than Boost.Build's.
>>
>> Here is cmake to implement simple compile fail testing where a regex on
>> the error messages from the compiler must be matched for the test to
>> pass:
>>
>> https://github.com/ned14/quickcpplib/blob/master/cmakelib/QuickCppLibMakeStandardTests.cmake#L142
>
> Thanks ! I will look at it. It does seem like a great deal of code just
> to run a bunch of tests.

Only four lines or so are needed to make a compile-fail test in cmake.
You can extract those out from my code easily enough.

The reason for all the code is that QuickCppLib automates everything.
You simply point it at a directory tree laid out in boost format, fill
in a few parameters like <library name>, and whoosh it all just works
with a complete package of support for docs, sanitiser tests, C++
Modules, precompiled headers, versioning, mangling, the whole kaboosh.

But that's cmake innovation. And I wouldn't recommend that for Boost.
You guys need something as simple and as minimal as possible which
matches, as close as possible, Boost.Build. Otherwise you'll never
finish converting over the build system, you'll get 60% there and that
long tail will become insurmountable given it's volunteer time. We saw
exactly the same happen with the git migration, it's that long tail
which kills you by causing people to quit the whole project rather than
put up with the never ending criticism.

I personally think that a build directory per configuration design is
too dissimilar to Boost.Build that such a conversion could ever be
finished without injecting paid death march hours. But I won't be doing
the work, so it's up to those who are to choose their design poison.

>> This cmake reads the regex from the first line of the fail test's source
>> code.
>>
>> There is lots of flexibility in cmake's fail test design. You can make
>> tests which must compile but must fail to link. You can make tests which
>> must complete within a certain time period. Lots of flexibility.
>>
>> ctest generates XML of all the test results. That can be merged with
>> JUnit XML from the individual tests easily enough. And the whole lot
>> sent up to CDash e.g.
>> http://my.cdash.org/index.php?project=Boost.Outcome&date=2017-07-23
>> where fails and successes can be drilled into, and the cause figured out.
>
> I have a feeling that CTest is not usable for the current Boost testing
> system, but maybe I am wrong.

CTest I think is more than able. The sole thing I've ever found it falls
badly down on is running tests on real remote devices e.g. Android ones.
It really wants to run on-device, not remotely. Still, it can be beaten
into submission using evil trickery.

CDash probably isn't up to Boost's requirements. But the XML CTest
outputs is standardised, it's not even Kitware's XML design. So the
tooling currently producing the Boost regression matrix etc ought to be
very easy to teach to understand CTest XML output, and thus all becomes
well.

Niall

-- 
ned Productions Limited Consulting
http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/

Boost list run by bdawes at acm.org, gregod at cs.rpi.edu, cpdaniel at pacbell.net, john at johnmaddock.co.uk