Skip to main content

Software engineering, checked · Part 13 of 13

What survives

November 19, 2026·7 min read
34 claims checked

Strip fifty years of software engineering folklore down to the claims that survive, and what's left is smaller than the conference talks promised and more useful than most of them delivered.

It isn't a methodology. Nothing in this series rated ~95%, and nobody who reads the evidence will come away believing in a process that fixes software. What remains is closer to a set of habits: at each step in the life of a change, a few things the evidence supports, and a longer list of things everyone repeats that it doesn't.

Every claim the series cites, one row per part.

The life of a change on one page

  • Decide. Assume the idea is a guess, and measure whether it worked. Best evidence: only about a third of the ideas tested at Microsoft moved the metric they were built for ~85 %.
  • Estimate. Expect about 30% over, plan for the tail, and estimate from history, independently, with wider ranges. Best evidence: the 30% average across decades of surveys ~85 %; independent estimates combined beat most individuals ~85 %; 90% ranges catch the truth two times in three ~85 %.
  • Organise. Make sure the people whose code depends on each other talk, and give code a clear owner. Best evidence: designs mirror organisations in most firms ~70 %; matching communication to dependencies speeds work ~70 %.
  • Write. Pick types for the bugs they catch and memory safety for the vulnerabilities it removes; don't expect the language to fix quality. Best evidence: a type checker would catch about 15% of public JavaScript bugs ~85 %; most serious vulnerabilities in big C and C++ codebases are memory-safety bugs ~85 %.
  • Review. Review to share understanding, and nudge the reviews that stall. Best evidence: three quarters of what review finds is maintainability, not bugs ~85 %; automatic reminders cut completion time sharply ~85 %.
  • Test. Test the error handlers, fuzz the inputs. Best evidence: most catastrophic distributed failures came from errors the software had already detected ~70 %; random input still crashes standard tools ~85 %.
  • Ship. Release often, fear the rush rather than the weekday, and make undo cheap. Best evidence: shorter release cycles didn't lower quality at Firefox ~70 %; most high-severity incidents ended without a code change ~70 %.
  • Age. Watch where code changes, and keep dependencies moving. Best evidence: change history predicts defects better than code measures ~85 %; most developers don't update dependencies, even after an advisory ~85 %.

Read the evidence column again. The strongest entries are an experiment platform's base rate, decades of overrun surveys, a count of bugs in public repositories, and the incident logs of a large service. None of them is a methodology. All of them are measurements of what actually happened, which is what most of the folklore never was.

Three lessons from the checking

Follow the number home. The field's most famous numbers turned out to be origin stories. The 100x cost of a production bug leads back to unpublished 1970s data ~30 %. The 189% overrun came from executives asked for failure stories ~15 %. The 28:1 programmer came from a dozen people in 1968 ~15 %. "60–90% of cost is maintenance" came from a survey's rough estimates. Even the waterfall is a number of sorts, a diagram whose author drew it to say it rarely works ~15 %. In every case the figure travelled and the caveat stayed home. A useful reading habit follows: when a talk quotes a precise number about software, ask what was measured, in how many projects, and when. Usually nobody knows.

Software is people talking. The findings that held up best were rarely about code. Systems copy the organisations that build them. Code with a clear owner breaks less, and code changed near a reorganisation breaks more ~70 %. Review's real product is shared understanding ~70 %. A reminder bot speeds up pull requests more reliably than making them small. Being watched cut interview performance by more than half ~70 %. Configuration, where one team's assumptions meet another team's system, keeps turning up in outages. Brooks made this argument in 1975. His numbers aged badly; his point is the best-supported idea in the series.

What engineers believe is not what gets measured. Developers think they spend half their time testing; their IDEs recorded about a quarter ~30 %. Experienced developers felt 20% faster with AI tools while the stopwatch said 19% slower ~85 %. The same expert re-estimating the same task disagreed with themselves by 71% ~70 %. Features that senior stakeholders were excited about kept failing their experiments ~15 %. Developers said slow builds drove them from CI; their projects' behaviour said otherwise. This is the productivity series' last lesson again, in an engineering costume: the instrument most teams steer by, their own sense of how it's going, is the least reliable one they have.

The hall of shame

Some ideas are worth knowing mainly so you can stop repeating them:

  • "A bug costs 100 times more in production." Old, unpublished data; 171 modern projects showed no consistent effect ~30 %.
  • "Only 16% of projects succeed." A definition that even perfect forecasts fail ~15 %.
  • "Iterative development was invented by agile." It goes back to the 1950s ~5 %.
  • "Some people can't learn to program, and a test can tell." The camel test was retracted by its author ~5 %.
  • "The language you choose decides how buggy your code is." The famous study shrank to almost nothing on reproduction ~15 %.
  • "Formal models beat expert estimates." Experts won more often ~15 %.
  • "Redundancy prevents outages." The failover chain is software too ~15 %.
  • "Minor and patch releases don't break your code." In Maven, over a third of minor releases broke the API ~15 %.

None of these were fringe ideas. Each appears in textbooks, talks or team rules, and most were stated with a precise number. A reasonable test for the next piece of software folklore is whether it would make this list in five years.

Where to start

Don't adopt everything at once. The life of a change is a line, and a line is only as fast as its slowest step. Find the step where your team hurts most. If releases are frightening, it's Ship: make undo cheap before anything else. If everything is late, it's Estimate. If nobody knows who understands the billing code, it's Organise and Review. Fix that, watch the results for a quarter rather than a sprint, then look for the next one.

A minimal version fits on an index card:

  1. Before building, write down what the change should move, and check afterwards.
  2. Estimate from past work, independently, and widen the range.
  3. Give every part of the code an owner, and make sure the people who depend on each other talk.
  4. Review to understand, and chase the reviews that stall.
  5. Test what happens when things fail, not only when they work.
  6. Make every release easy to undo.
  7. Measure before you believe, especially about your tools.

A version number

Part 1 started with a leprechaun and a reminder that careful people get things wrong. This series will be wrong in places too. The ratings are judgements, the selection was one person's, and some findings, the AI studies above all, will look different within a year.

So treat this as version 1. Every claim here is rated so that it can be re-rated. When the evidence moves, the verdicts will move with it, and the short list will be rebuilt from whatever is still standing. That, after all, is the third lesson applied to the series itself: trust the measurement, not the feeling that it's finished.

34 claims checked

The evidence behind this article

Every claim above, rated by the chance it holds up against the research. Open one for the sources and their exact words.

  1. Bacchelli, Bird (2013). Expectations, outcomes, and challenges of modern code review · Study
  2. Behroozi, Parnin, Barik (2019). Hiring is Broken: What Do Developers Say About Technical Interviews? · Study
  3. Behroozi, Shirolkar, Barik, Parnin (2020). Does stress impact technical interview performance? · Study
  4. Beller, Bacchelli, Zaidman, Juergens (2014). Modern code reviews in open-source projects: which problems do they fix? · Study
  5. Beller, Gousios, Panichella, Proksch, Amann, Zaidman (2019). Developer Testing in the IDE: Patterns, Beliefs, and Behavior · Study
  6. Beller, Gousios, Panichella, Zaidman (2015). When, how, and why developers (do not) test in their IDEs · Study
  7. Berger, Hollenbeck, Maj, Vitek, Vitek (2019). On the Impact of Programming Languages on Code Quality · Replication
  8. Boehm, Basili (2001). Top 10 list [software development] · Source
  9. Cataldo, Herbsleb (2013). Coordination Breakdowns and Their Impact on Development Productivity and Software Failures · Study
  10. Cataldo, Wagstrom, Herbsleb, Carley (2006). Identification of coordination requirements · Study
  11. Colfer, Baldwin (2016). The mirroring hypothesis: theory, evidence, and exceptions · Source
  12. Dickey (1981). Programmer variability · Source
  13. Eveleens, Verhoef (2010). The rise and fall of the Chaos report figures · Study
  14. Flyvbjerg, Budzier (2013). Why Your IT Project Might Be Riskier Than You Think · Study
  15. Foucault, Palyart, Blanc, Murphy, Falleri (2015). Impact of developer turnover on quality in open-source software · Study
  16. Gao, Bird, Barr (2017). To Type or Not to Type: Quantifying Detectable Bugs in JavaScript · Study
  17. Ghosh, Shetty, Bansal, Nath (2022). How to fight production incidents? · Study
  18. Graves, Karr, Marron, Siy (2000). Predicting fault incidence using software change history · Study
  19. Grimstad, Jørgensen (2007). Inconsistency of expert judgment-based estimates of software development effort · Study
  20. Gunawi, Hao, Suminto, Laksono, Satria, Adityatama, Eliazar (2016). Why Does the Cloud Stop Computing? · Study
  21. Haskins, Stecklein, Dick, Moroney, Lovell, Dabney (2004). 8.4.2 Error Cost Escalation Through the Project Life Cycle · Source
  22. Hassan (2009). Predicting faults using the complexity of code changes · Study
  23. Haßler, Görz, Lipp (2025). A Comparative Study of Fuzzers and Static Analysis Tools for Finding Memory Unsafety in C and C++ · Study
  24. Joel, Rush, Barnes, David (2025). Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity · Study
  25. Jorgensen (2014). What We Do and Don't Know about Software Development Effort Estimation · Study
  26. Jørgensen (2004). A review of studies on expert estimation of software development effort · Meta-analysis
  27. Jørgensen (2007). Forecasting of software development work effort: Evidence on expert judgement and formal models · Meta-analysis
  28. Jørgensen, Moløkken (2004). Eliminating Over-Confidence in Software Development Effort Estimates · Study
  29. Jørgensen, Moløkken-Østvold (2006). How large are software cost overruns? A review of the 1994 CHAOS report · Study
  30. Khomh, Adams, Dhaliwal, Zou (2014). Understanding the impact of rapid releases on software quality · Study
  31. Khomh, Dhaliwal, Zou, Adams (2012). Do faster releases improve software quality? An empirical case study of Mozilla Firefox · Study
  32. Kohavi, Deng, Frasca, Walker, Xu, Pohlmann (2013). Online controlled experiments at large scale · Study
  33. Kohavi, Longbotham, Sommerfield, Henne (2008). Controlled experiments on the web: survey and practical guide · Study
  34. Kula, German, Ouni, Ishio, Inoue (2017). Do developers update their library dependencies? · Study
  35. Kwan, Schroter, Damian (2011). Does Socio-Technical Congruence Have an Effect on Software Build Success? A Study of Coordination in a Software Project · Study
  36. Larman, Basili (2003). Iterative and incremental developments. a brief history · Source
  37. Lung, Aranda, Easterbrook, Wilson (2008). On the difficulty of replicating human subjects studies in software engineering · Replication
  38. MacCormack, Baldwin, Rusnak (2012). Exploring the duality between product and organizational architectures: A test of the “mirroring” hypothesis · Study
  39. Maddila, Upadrasta, Bansal, Nagappan, Gousios, van Deursen (2023). Nudge: Accelerating Overdue Pull Requests toward Completion · Study
  40. Mantyla, Lassenius (2009). What Types of Defects Are Really Discovered in Code Reviews? · Study
  41. Menzies, Nichols, Shull, Layman (2016). Are delayed issues harder to resolve? Revisiting cost-to-fix of defects throughout the lifecycle · Study
  42. Miller, Fredriksen, So (1990). An empirical study of the reliability of UNIX utilities · Study
  43. Miller, Zhang, Heymann (2022). The Relevance of Classic Fuzz Testing: Have We Solved This One? · Replication
  44. Mockus (2010). Organizational volatility and its effects on software defects · Study
  45. Molokken, Jorgensen (2003). A review of software surveys on software effort estimation · Meta-analysis
  46. Prechelt (2000). An empirical comparison of seven programming languages · Study
  47. Raemaekers, van Deursen, Visser (2017). Semantic versioning and impact of breaking changes in the Maven repository · Study
  48. Rahman, Devanbu (2013). How, and why, process metrics are better · Study
  49. Ray, Posnett, Filkov, Devanbu (2014). A large scale study of programming languages and code quality in github · Study
  50. Rigby, Bird (2013). Convergent contemporary software peer review practices · Study
  51. Sackman, Erikson, Grant (1968). Exploratory experimental studies comparing online and offline programming performance · Study
  52. Sadowski, Söderberg, Church, Sipko, Bacchelli (2018). Modern code review · Study
  53. Thongtanunam, McIntosh, Hassan, Iida (2015). Investigating Code Review Practices in Defective Files: An Empirical Study of the Qt System · Study
  54. Yuan, Luo, Zhuang, Rodrigues, Zhao, Zhang, Jain, Stumm (2014). Simple Testing Can Prevent Most Critical Failures: An Analysis of Production Failures in Distributed Data-Intensive Systems · Study
  55. Zimmermann, Staicu, Tenny, Pradel (2019). Small World with High Risks: A Study of Security Threats in the npm Ecosystem · Study

Share

XLinkedIn