Re: Code coverage tests

From: Hans Dembinski (hans.dembinski_at_[hidden])
Date: 2020-04-14 09:22:12

> On 13. Apr 2020, at 18:54, Vinnie Falco via Boost <boost_at_[hidden]> wrote:
> 
> I have adopted a
> style of code that helps the code coverage report make more sense. For
> example by arranging conditionals to make it more clear when branches
> are taken versus not. And I have tweaked the coverage command line
> settings to eliminate false negatives. The coverage reports have been
> incredibly successful in helping me identify which parts of code need
> better test coverage [...]

I had 99 % line coverage for a while in Boost.Histogram, eventually I got it to 100 %. The Pareto principle applies, getting 80 % requires 20 % of the work, while the remaining 20 % required 80 % of the work. To get the last 1 %, I had to write a mocked allocator that throws std::bad_alloc in a controlled way, and finally I also had to add some LCOV_EXCL_LINE, LCOV_EXCL_START, LCOV_EXCL_STOP, because gcc-8 reports some lines as missed which cannot possibly be missed and which gcc-5 did report correctly.

My points:

1) I agree with Vinnie that achieving line coverage close to 100 % is a worthy goal, and yes that requires also to understand how the coverage reporting works. I found many bugs in my code this way, including some subtle ones. I cannot guarantee that the code is bug-free, but I sleep rather well at night. Maybe we should write a guidelines about this to share what we learned.

2) Getting branch coverage to 100 % is nearly impossible with reasonable effort (I consider the effort I put into getting 100 % line coverage already barely reasonable), and that's why I am in the camp with Vinnie that it is not worth it. Consider this code

if (cond1 && cond2 && cond3)
   // do X
else
   // do Y

Branch coverage will insist that you check all(*) combinations of true/false for cond1, cond2, and cond3 for 100 %, while in most cases, it really only matters that both "do X" and "do Y" are executed in tests and work correctly.

(*) It may detect the short-circuiting of the "&&", I am not sure. So maybe you don't have to test all combinations, but at least four (TTT, FXX, TFX, TTF).

3) Achieving nearly full coverage is much easier for libraries that do not have to support workarounds for old compilers. It is considerably easier to achieve for the C++11 and C++14 libraries than for the older ones. Perhaps older Boost libraries should not attempt this.

Best regards,
Hans