Skip to main content

Software engineering, checked · Part 7 of 13

Test: the failures we could have prevented

November 19, 2026·10 min read
20 claims checked

In 2014, Ding Yuan and colleagues collected 198 failures that users had reported in five widely used distributed systems: Cassandra, HBase, HDFS, Hadoop MapReduce and Redis. These are systems designed to survive disks dying and machines disappearing. The team looked closely at the failures that were catastrophic. In 92% of them, the software had already noticed the problem. A non-fatal error had been signalled, explicitly, and the code meant to handle it handled it badly ~70 %.

Many of those handlers weren't subtle. In 35% of the catastrophic failures the fault sat in a trivial pattern, such as an error handler that was empty or did nothing but print a log line. In 58%, the authors concluded, the fault "could easily have been detected through simple testing of error handling code", even by someone who didn't understand the system's design ~70 %.

Software engineering's usual case for testing is about timing: a bug costs more the later you find it, the hundredfold curve that part 1 traced back to 1970s data nobody can check. Yuan's study makes a different and more useful case. It is about where the bugs are, not when they are found. The question is plain: which tests prevent the failures that matter, and do the practices we argue about deliver them?

Where the failures hide

Yuan's paper had a second piece of good news for testers. Almost all of the failures could be reproduced on three nodes or fewer, although these services typically run on very many ~70 %. You don't need a production-sized cluster to find them. The catch is order: most needed several inputs, and the sequence in which they arrived mattered. A test that sends one request and checks one answer walks straight past them.

Should empty catch blocks be banned, then? It depends where they sit. A study of exception-handling bugs in Eclipse and Tomcat found that overly general catch blocks rarely caused reported bugs, and that empty ones often turned up as part of the fixes ~50 %. In a system whose whole job is to keep running when parts of it fail, the error handler is, in Yuan's phrase, the last line of defence.

Failures also cluster in code. Norman Fenton and Niclas Ohlsson found, in a large telecom system, that a small number of modules contained most of the faults found in testing, and a very small number contained most of those found in operation. Two replications confirmed the uneven spread, and a summary by Barry Boehm and Victor Basili put it at 60 to 90 percent of defects in 20 percent of the modules, with a median of about 80 ~85 %. Neil Walkinshaw and Leandro Minku checked 100 GitHub projects and found the 80:20 split holds, but only if a fix that touches several files counts as several defects. Either way, testing effort spread evenly across a codebase is spread thinly where the bugs are.

Where does 40–90% come from?

Test-driven development has a famous number of its own: TDD cuts defects by 40 to 90 percent. It comes from a 2008 paper by Nachiappan Nagappan and colleagues, who followed four industrial teams at Microsoft and IBM. The pre-release defect density of the four products fell by between 40% and 90% compared with similar projects that didn't use TDD ~30 %.

That is a real result, and a narrow one. These were four case studies, each compared with projects the authors judged similar, not controlled trials. Whatever else differed between the TDD teams and their comparisons is folded into the figure.

When researchers pooled the studies, the effect shrank. A 2013 meta-analysis of 27 studies found that TDD had a small positive effect on quality and little to no effect on productivity. Controlled experiments went further. An external replication across several sites, analysed blind, found no significant difference between TDD and test-last development in testing effort, code quality or productivity. A family of twelve experiments found that TDD novices wrote slightly better code with iterative test-last development, TDD in reverse ~50 %.

The cost is just as unclear. The Microsoft and IBM teams felt that their initial development took 15 to 35 percent longer, a subjective figure. The meta-analysis found both the quality gain and the productivity drop much larger in industrial studies than in academic ones, and the blind replication found no significant difference in productivity ~50 %. The honest summary is that nobody yet knows whether TDD, on its own, makes code better or teams slower.

Test first, or small steps?

If TDD helps, the obvious explanation is the one in its name: write the test first. Davide Fucci and colleagues took the process apart in a study of 39 professionals, separating the order in which tests and code were written from the size and rhythm of the steps. The order had no important influence. The benefits, they suggested, come from the "fine-grained, steady steps" that TDD-like processes encourage, which "improve focus and flow" ~30 %.

That moves the argument. Red, green, refactor may matter less than what the ritual enforces: short loops, each one checked before the next begins. Writing a small piece of code and then its test straight away seems to do about as well as writing them the other way round.

Grading the tests

Most teams that measure their tests measure coverage, and coverage is a weaker signal than it looks. In the largest study, by Laura Inozemtseva and Reid Holmes, the correlation between coverage and a suite's effectiveness was low to moderate once the number of tests was controlled for. With real bugs in large systems, another study did find a significant correlation, though it varied from system to system, and a third found that fault detection only rises markedly at relatively high coverage ~50 %.

Fancier coverage doesn't help. Branch, block and path coverage gave no more insight than plain statement coverage, which in one study predicted test quality best of the four ~15 %. The one criterion that did better was mutation.

Mutation testing makes small deliberate changes to the code, such as flipping a comparison or deleting a line, and checks whether any test notices. Each change is a mutant; a suite that lets mutants survive has gaps. René Just and colleagues found that catching mutants correlates with catching real faults, independently of coverage. A later study found the correlation weak once suite size is controlled, while still recommending mutants as good guidance for improving a suite ~70 %. Mutation testing was long thought too expensive for real codebases, because the number of possible mutants is enormous. Google now runs it on the changed code during code review, for more than 24,000 developers on more than 1,000 projects ~15 %.

What developers think they test

Fred Brooks's old rule of thumb gave half of a project's schedule to testing, and developers still believe something like it. Moritz Beller and colleagues monitored engineers' IDEs, first 416 of them and then 2,443 over two and a half years. The developers thought they spent half their time testing. Measured, it was about a quarter ~30 %.

The same data showed more. Half of the developers didn't test at all in their IDE, they rarely ran tests there, and most programming sessions ended without a single test execution ~70 %. One caveat matters: the study saw only the IDE. Tests that ran in a CI pipeline were invisible to it, so "half don't test" says more about habits at the keyboard than about what reaches production.

Some of the tests that do run can't be trusted. In Beller's data, 12% of tests showed flaky behaviour, passing and failing without any change to the code, and a quarter of the tests caused three quarters of all failures. In a survey of developers, 79% called flaky tests a moderate to serious problem, and 40% dealt with them at least weekly ~85 %. A suite that fails at random teaches its readers to press "rerun" instead of reading the failure. Once a team stops believing its tests, the tests stop doing their job, however good they are.

Machines that test

In 1990, Barton Miller and colleagues fed random input to almost 90 utility programs on seven versions of Unix and crashed more than 24% of them. The experiment started fuzzing as a field. When they ran it again three decades later, on Linux, FreeBSD and macOS, it still crashed or hung 9 of 74, 15 of 78 and 12 of 76 utilities ~85 %. Thirty years on, random typing remained an effective test.

Static analysis has a bigger reputation than its record. Three widely used Java bug detectors, run against 594 real bugs, found a non-negligible share, complemented each other, and missed the large majority ~15 %. At Google, engineers in a week-long fix-it reviewed 3,954 FindBugs warnings, filed bug reports for 44% of them, and fixed 16%. The problem was rarely the analysis itself. Developers lose confidence in tools that raise false alarms, and Google's bug dashboard saw little use because it sat outside the developers' usual workflow. Static analysis took hold at Google once its checks lived in the compiler and in code review, where developers were already talking about the change ~85 %.

A narrow check aimed at a known pattern can still be worth a great deal. Yuan's team built a static checker of their own, Aspirator. Over 30% of the catastrophic failures, they estimated, would have been prevented had it been used and its findings fixed.

The proof that paid

Formal methods have the opposite reputation: rigorous, expensive and only worth it for aircraft and medical devices. In 2015, Chris Newcombe and colleagues described how Amazon Web Services used TLA+, a language for specifying and checking system designs, on ten large systems. It found subtle bugs that nothing else had found. Engineers "from entry level to Principal" learned it from scratch and got useful results in two to three weeks. The reputation, they wrote, was "quite wrong" ~30 %.

That is one company's experience report, written by enthusiasts. A broader survey of formal-methods projects leans the same way, but in most of the projects surveyed there was no data on time and cost to compare. The case is strongest where Yuan found the worst failures: distributed designs, where a bug needs several events in a particular order, and ordinary tests rarely look.

What doesn't work

  • Quoting TDD's 40–90%. It comes from four case studies, not trials; pooled, the effect on quality is small ~30 %.
  • Treating test-first as the active ingredient. The order of test and code made no important difference; the size of the steps did ~30 %.
  • Grading suites by branch or path coverage. The stronger criteria told researchers no more than statement coverage did ~15 %.
  • Relying on static analysis to catch most bugs. The best-known tools missed the large majority of real ones ~15 %.
  • Trusting your sense of how much you test. Developers think half their time; measured, it was a quarter ~30 %.

Try this

  • Test the error handlers. For each catch block in code that must stay up, write the test that makes it run: the timeout, the full disk, the refused connection. Treat an empty or log-only handler as a review question.
  • Test small, in many orders. Three nodes are usually enough to reproduce a distributed failure; what they need is varied sequences of inputs, not more machines.
  • Spend testing where the bugs were. Your own bug history shows which modules hold most of the faults. Start there.
  • Use coverage to find untested code, not to grade tests. If you want a grade, run mutation testing on the changed code during review.
  • Put the machines in the conversation. Fuzz anything that parses input, surface static-analysis findings in the compiler or the review, and fix or delete flaky tests rather than rerunning them.

20 claims checked

The evidence behind this article

Every claim above, rated by the chance it holds up against the research. Open one for the sources and their exact words.

  1. Andersson, Runeson (2007). A Replicated Quantitative Analysis of Fault Distributions in Complex Software Systems · Replication
  2. Beller, Gousios, Panichella, Proksch, Amann, Zaidman (2019). Developer Testing in the IDE: Patterns, Beliefs, and Behavior · Study
  3. Beller, Gousios, Panichella, Zaidman (2015). When, how, and why developers (do not) test in their IDEs · Study
  4. Boehm, Basili (2001). Top 10 list [software development] · Source
  5. Chekam, Papadakis, Le Traon, Harman (2017). An Empirical Study on Mutation, Statement and Branch Coverage Fault Revelation That Avoids the Unreliable Clean Program Assumption · Study
  6. Ebert, Castor, Serebrenik (2015). An exploratory study on exception handling bugs in Java programs · Study
  7. Eck, Palomba, Castelluccio, Bacchelli (2019). Understanding flaky tests: the developer’s perspective · Study
  8. Fenton, Ohlsson (2000). Quantitative analysis of faults and failures in a complex software system · Study
  9. Fucci, Erdogmus, Turhan, Oivo, Juristo (2016). A Dissection of the Test-Driven Development Process: Does It Really Matter to Test-First or to Test-Last? · Study
  10. Fucci, Scanniello, Romano, Shepperd, Sigweni, Uyaguari, Turhan, Juristo, Oivo (2016). An External Replication on the Effects of Test-driven Development Using a Multi-site Blind Analysis Approach · Replication
  11. Galinac Grbac, Runeson, Huljenic (2013). A Second Replicated Quantitative Analysis of Fault Distributions in Complex Software Systems · Replication
  12. Gopinath, Jensen, Groce (2014). Code coverage for suite evaluation by developers · Study
  13. Habib, Pradel (2018). How many of all bugs do we find? a study of static bug detectors · Study
  14. Inozemtseva, Holmes (2014). Coverage is not strongly correlated with test suite effectiveness · Study
  15. Johnson, Song, Murphy-Hill, Bowdidge (2013). Why don't software developers use static analysis tools to find bugs? · Study
  16. Just, Jalali, Inozemtseva, Ernst, Holmes, Fraser (2014). Are mutants a valid substitute for real faults in software testing? · Study
  17. Kochhar, Thung, Lo (2015). Code coverage and test suite effectiveness: Empirical study with real bugs in large systems · Study
  18. Miller, Fredriksen, So (1990). An empirical study of the reliability of UNIX utilities · Study
  19. Miller, Zhang, Heymann (2022). The Relevance of Classic Fuzz Testing: Have We Solved This One? · Replication
  20. Nagappan, Maximilien, Bhat, Williams (2008). Realizing quality improvement through test driven development: results and experiences of four industrial teams · Study
  21. Newcombe, Rath, Zhang, Munteanu, Brooker, Deardeuff (2015). How Amazon web services uses formal methods · Study
  22. Papadakis, Shin, Yoo, Bae (2018). Are mutation scores correlated with real fault detection? · Study
  23. Parry, Kapfhammer, Hilton, McMinn (2021). A Survey of Flaky Tests · Meta-analysis
  24. Petrovic, Ivankovic, Fraser, Just (2022). Practical Mutation Testing at Scale: A view from Google · Study
  25. Rafique, Misic (2013). The Effects of Test-Driven Development on External Quality and Productivity: A Meta-Analysis · Meta-analysis
  26. Sadowski, Aftandilian, Eagle, Miller-Cushon, Jaspan (2018). Lessons from building static analysis tools at Google · Study
  27. Santos, Vegas, Dieste, Uyaguari, Tosun, Fucci, Turhan, Scanniello, Romano, Karac, Kuhrmann, Mandić, Ramač, Pfahl, Engblom, Kyykka, Rungi, Palomeque, Spisak, Oivo, Juristo (2021). A family of experiments on test-driven development · Study
  28. Walkinshaw, Minku (2018). Are 20% of files responsible for 80% of defects? · Study
  29. Woodcock, Larsen, Bicarregui, Fitzgerald (2009). Formal methods · Meta-analysis
  30. Yuan, Luo, Zhuang, Rodrigues, Zhao, Zhang, Jain, Stumm (2014). Simple Testing Can Prevent Most Critical Failures: An Analysis of Production Failures in Distributed Data-Intensive Systems · Study

Share

XLinkedIn