A Philosophy of Testing 5: The Case for 100%

What is “Coverage”?

Not everyone is familiar with coverage tools, so terminology first: for purposes of this series, “coverage” refers to automated tools, generally integrated into sbt and invoked at build time, that check how comprehensively your tests “cover” your code. (scoverage and its variations are popular ways to do this in Scala.)

lazy val rootProject = (project in file("."))
.settings(coverageMinimumStmtTotal := 100.0)
.settings(coverageFailOnMinimum := true)

Why 100%?

This topic is a frequent source of arguments, and some folks get their backs up about the concept of requiring 100% coverage — it’s very common to say that, say, 80% is fine and much more realistic.

“100%” Doesn’t Really Mean 100%

That’s where we need to start getting into nuance, though. I would strongly recommend that you tell sbt to check 100% of your code. But that doesn’t actually mean you need to be checking every line of code.

if (thingAreGoingAsExpected) {
doTheHappyPath()
} else {
// I believe this code path is impossible to hit,
// because...
// $COVERAGE-OFF$
reportAnError()
// $COVERAGE-ON$
}

When and How to Exempt Code

There are several situations in which it is legit and normal to turn coverage off.

Untestable Scala

First and simplest, any final val lines like this are fundamentally untestable:

final val myConstant = "Hello world"
if (thingAreGoingAsExpected) {
doTheHappyPath()
} else {
// I believe this code path is impossible to hit,
// because...
// $COVERAGE-OFF$
reportAnError()
// $COVERAGE-ON$
}

Not Tested Yet

The most common situation is where you have written some code that is mostly tested, but which has an edge case that is both uncommon and annoyingly hard to test. In theory, you should of course test everything before merging that code; in practice, the pressures of real-world deadlines often don’t permit that.

if (thingAreGoingAsExpected) {
doTheHappyPath()
} else {
// TODO (FOO-123): come back and test this
// $COVERAGE-OFF$
reportAnError()
// $COVERAGE-ON$
}

100% Isn’t Enough

Don’t fool yourself into believing that, just because you have 100% coverage, your code is necessarily bug-free: it absolutely does not prove that. You still need to make sure that you have a test for each likely use case. In practice, any time you make an enhancement, that should be represented somewhere in the tests.

Death to Dead Code!

Finally — your 100% test coverage will often reveal (especially when you are initially getting to that 100%) that you have a bunch of code that isn’t tested because it really, truly isn’t being used. The top-level entry points of your service simply don’t lead there.

Getting There From Here

All of the above is talking mainly about the ideal end-state. But say that you have a big code base already, and haven’t done a lot of testing yet. You install scoverage — and it tells you that your coverage is 3%. It’s very easy to get depressed about that, and just write the whole thing off.

Summary

I hope you’ll seriously consider the above, and think about applying it to your own applications. 100% coverage is the only philosophically honest level to set; when you couple that with disciplined exceptions, it’s entirely practical. It helps you set a minimum level for your tests, which will help you build reliable systems.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Mark "Justin" Waks

Mark "Justin" Waks

Lifelong programmer and software architect, specializing in online social tools and (nowadays) Scala. Architect of Querki (“leading the small data revolution”).