Re: Code coverage tests

From: Vinnie Falco (vinnie.falco_at_[hidden])
Date: 2020-04-13 16:54:53

On Mon, Apr 13, 2020 at 9:44 AM Edward Diener via Boost
<boost_at_[hidden]> wrote:
> Maybe code coverage is not all it is cracked up to be <g>. Sorry, but I
> distrust something that is so unsubtle.

Code coverage is simply tool, and like all good things requires effort
and attention to detail to use correctly. Depending on how the code is
structured, yes the tool will claim a lack of coverage if the
particular toolchain used to run the tests does not exercise a code
path used for a separate work-around. There are of course ways to
mitigate this. You can compile using both toolchains, and run the
resulting tests, and accumulate the coverage reports into one merged
report which does include the workaround code.

Another solution is to express the workaround code using function or
class templates. These don't count as uncovered lines unless they are
instantiated. Yes, I realize that this could be considered a "trick"
(i.e. sleight of hand) but as I said, the code coverage is a tool that
requires diligence. One cannot expect to just throw a code base at a
coverage tool and get perfect results. You have to pick through it and
identify things such as the case you described, and make small
adjustments and fixes. Ultimately you need to decide if and how the
coverage tool can enhance your project, and adopt a workflow that
makes the reports useful.

This is the approach that I have taken with Boost.Beast,
Not-Yet-In-Boost.JSON, and Not-Yet-In-Boost.URL. I have adopted a
style of code that helps the code coverage report make more sense. For
example by arranging conditionals to make it more clear when branches
are taken versus not. And I have tweaked the coverage command line
settings to eliminate false negatives. The coverage reports have been
incredibly successful in helping me identify which parts of code need
better test coverage, giving me a high degree of confidence that the
code in question is without defects. Boost.JSON for example has almost
99% coverage (I had a big refactor recently which caused a few new
uncovered lines, otherwise it would be over 99%):

<https://codecov.io/gh/vinniefalco/json/list/develop/>

Thanks