We all love metrics in the startup world, and tend to have a strange relationship with them. This added with the Engineering minds of CTO’s, VP’s and Dev Managers, and you’ve got the perfect metric trap – Code Coverage.

Code Coverage might look great for a bit, but trust me on this one… it’s a trap.

The Code Coverage Trap.

Admiral Ackbar: “Take evasive action. Green Group, stay close to holding sector MD-7.”
Crewman: “Admiral, we have enemy ships in sector 47!”
Ackbar: “It’s a trap!”

So What Is Code Coverage?

Wikipedia has a decent definition.

Code Coverage is a measure used to describe the degree to which the source code of a program is tested by a particular test suite… Many different metrics can be used to calculate code coverage; some of the most basic are the percent of program subroutines and the percent of program statements called during execution of the test suite.

In my experience statement or line coverage is the most common type of code coverage goal Dev Teams set. With line coverage you’re trying to make sure that statement of code is “touched” by test cases.

This is normally measured in percentage 80%, 90%, 100% etc.

So now that we’ve got that out-of-the-way.

Flaws With Focusing Only On Code Coverage

It all boils down to this one truth. You can have 100% code coverage and still have buggy software.

Let say we want to return x if x and y are less than 0. In any other case you’ll want to return -1.

java Buggy Code
public int foo (int x, int y) {
int z = 0;
if ((x > 0) && (y > 0)) {
z = x;
}
return z;
}

With the code above we could have complete statement coverage with 1 test calling foo(1,1). You still wouldn’t catch the bug though.

Exponential growth curve in difficulty as you try to reach upper percentages in coverage.

Have you ever noticed how the difference between amateur athletes can very dramatically. Huge gaps are seen between the winners and loser performed.

In the Olympics the time difference between a gold medalist and the other top places is very small. That’s because the closer you get to perfection the harder it can be to improve.

An increase of just 1% when you’re almost at the peak can take a lot of effort. This is true for your code coverage goals as well, and something else you’ll have to watch out for to avoid the trap.

A few other issues are:

• It ignores data or user input coverage completely.
• High percentage can be gained by only testing “happy path” scenarios.

This can all lead to a false sense of stability.

Avoiding The Trap

So you’re probably wondering “If I shouldn’t focus on Code Coverage then what should we do?”

Don’t worry I’ve got you covered. The idea is to gain code coverage without focusing on code coverage.

Code Coverage is just a tool. It’s a thermometer, that can be used to quantify a symptom. If you focus on testing all areas of your application then you’ll eventually get a high code coverage. You can then us the thermometer to check you temperature from time to time.

There’s nothing wrong with measuring your code coverage. It can help you identify areas of your application that might need a little bit more love. Problems come up when it because the end all be all for the team. This can be avoid though with a little effort and planning before hand.

The public knows that human beings are fallible. Only people blinded by ideology fall into the trap of believing in their own infallibility. -Freeman Dyson