100% code coverage is false safety

It is really easy to lie to yourself when doing Test Driven Development that your code is good, safe, predictable and well understood, just because you have that magic 100% code coverage number.
If you test just the happy case it is easy to get to 100% code coverage, while still leaving code paths untested and unexpected behaviour unexplored.

Consider this simple function that takes in an object with 2 properties, an array of strings and a separating string to concatenate between them.

1:  export const concatStrings = ({ strings, joiner }) => {  
2:     return strings.join(joiner);  
3:  };  

Testing the Happy Path is really easy.

1:  import { concatStrings } from "./stringConcat";  
2:  describe(concatStrings, () => {  
3:     it("Should concatinate strings", () => {  
4:        const actual = concatStrings({ joiner: ", ", strings: ["one", "two"] });  
5:        expect(actual).toBe("one, two");  
6:     });  
7:  });  

And with that one test we get 100% code coverage.


The problem is that while the coverage is 100% we have gamed the system to get tyere by passing in what we expect and testing the happy path case. However there are many unexpected code paths that are not covered by these tests despite the fact that Jest is reporting 100% coverage.

So What's not covered? The cases where the caller passes:

1. null or undefined as a parameter to the method.
2. an object with an unexpected shape to the method.
3. undefined or null as either of the properties on the parameter
4. unexpected object/data types as properties.

In part this is because of JavaScript's dynamic nature. A strongly typed complier would catch some of these cases, but the same issues could be present in a type safe language.

While this is very trivial example it is one that can quickly show how 100% code coverage not a guarantee that a piece of code is error free. In fact it doesn't even show that all the code paths are covered. Only that the code paths that have been explicitly written by the developer are covered. If you look at the points above many of them result in errors being thrown by the code. This code path isn't written so it's not counted in the coverage but it is none the less a path through this code. Similarly the behaviour around the null and undefined property paths are not explicitly written paths in the code so coverage doesn't show up that they're not explored.

More than 30 years ago Dijkstra wrote:

"Program testing can be a very effective way to show the presence of bugs, but it is hopelessly inadequate for showing their absence." - Edsger W. Dijkstra 1972

Take from this that the goal of testing, be it exploratory or automated, should be to actively find bugs. To do this better with unit tests the tests can be written to target the ways we expect the piece of code to fail first and only as a last resort write a test for the success case.

This does mean that you end up with more tests, and that the code takes longer to be done. The result though is code that is that much better understood. In the end, as always, it's a trade off. Finally remember that while 100% coverage doesn't guarantee that all the cases are covered 0% guarantees that all the cases are NOT covered.