Skip to main content

Software engineering, checked · Part 1 of 13

The leprechaun hunt

November 19, 2026·6 min read
9 claims checked

Every software engineer has seen the chart. A bug found during requirements costs one unit to fix. Found in design, five. In testing, perhaps twenty. Found in production, a hundred. The curve climbs like a ski jump, and the lesson writes itself: catch problems early, because later is ruinous. It appears in textbooks, conference talks, consulting decks and, at some point, in an argument near you.

In 2016, Tim Menzies and colleagues went looking for where it came from. The popular sources behind the curve, with more than 14,500 citations between them, all lead back to a single book: Barry Boehm's Software Engineering Economics, published in 1981 and built on late-1970s project data from IBM, TRW, GTE and Bell Labs. That data was never published for anyone to check, and it can no longer be found ~85 %. Boehm himself later wrote that for small, non-critical systems the factor is nearer 5:1 than 100:1. And when Menzies's team measured 171 modern projects, they found no consistent delayed-issue effect at all ~30 %.

Laurent Bossavit has a name for numbers like this: leprechauns. Everyone has heard of them, people describe them in detail, and nobody has actually seen one.

If the most-quoted number in software engineering is a leprechaun, the obvious question is how many others there are. This series goes hunting. It takes the field's famous books, from The Mythical Man-Month to Accelerate, and its folklore, from the 10x programmer to "never deploy on a Friday", breaks them into individual claims, and checks each one against the research.

How the claims were checked

Each claim gets a rating: the estimated chance that the claim, as it is usually stated, holds up against the evidence today. The scale is the same seven steps used in Productivity, checked:

  • ~95%: mathematics, or many large replications. You can build on it.
  • ~85%: strong evidence with minor caveats about size or generality.
  • ~70%: the direction is right, but the claim is bigger, broader or more causal than the evidence.
  • ~50%: genuinely open, mixed, or plausible but untested.
  • ~30%: weak, or partly contradicted.
  • ~15%: mostly contradicted, or a key study failed to replicate.
  • ~5%: failed replications, retracted, or conceded by the author.

A rating sits right after the sentence that makes the claim, like a citation. Open it for the verdict, the sources and the exact quotes behind it. Every quote was checked word for word against the paper's abstract or full text; a claim whose only evidence was a book, a vendor report or a paywall stayed out, and each part says which famous claims couldn't be checked. The ratings live in a claim library and are re-rated when the evidence moves, so a rating always shows the current verdict. They are judgement calls, not measurements.

What came out

283 claims, across ten areas of the field. About one in six rates ~85% or higher. More than a third sit at ~70%: right direction, wrong size. About one in seven is genuinely open. And nearly a third are weak or worse.

One number is missing: not a single claim reached ~95%. In the productivity series, the strongest entries were mathematics, like queueing theory, and findings replicated for decades, like the planning fallacy ~95 %. Software engineering has almost none of either. Its evidence is mostly single companies, open-source repositories mined after the fact, student experiments and surveys of what developers believe. Replications are rare, and when they happen they often shrink the result. Part 12 is about why.

That doesn't make the field's wisdom worthless. Some of it holds remarkably well. In 1986, Fred Brooks predicted that no single technology would deliver a tenfold gain in productivity within a decade, and four decades and one AI boom later, nothing has ~85 %.

Every claim the series cites, one row per part.

Three threads

Three patterns run through every part.

The famous numbers are origin stories, not measurements. The 100x curve is the first of many. "The best programmers are 28 times better than the worst" comes from a 1968 experiment with a dozen people ~15 %. "The average project overruns its budget by 189%" comes from a consultancy survey that asked executives for failure stories ~15 %. "Maintenance is 60–90% of the cost" goes back to a 1978 survey that quoted rough estimates ~70 %. The numbers travelled; the caveats stayed behind. Each part has a section on where its famous number came from.

Software is people talking. The strongest findings in this series are rarely about algorithms or languages. They are about who talks to whom, who knows what, and who is watching. Brooks made this argument in The Mythical Man-Month fifty years ago. His numbers have aged badly; his point hasn't.

What engineers believe is not what gets measured. Developers estimate they spend half their time testing; measured in their IDEs, it was about a quarter ~30 %. Experienced developers felt about 20% faster with AI tools while the stopwatch said they were 19% slower ~85 %. Over and over, the field's sense of how things are going and the measurements disagree, and the measurements usually win.

The life of a change

The series follows one piece of software from idea to old age and stops at each step where folklore gathers:

  1. Decide. Do we know what to build? Most ideas fail, and the waterfall wasn't what you were told.
  2. Estimate. Why software is late, and what the 189% really means.
  3. Organise. Conway's law, Brooks's law and the shape of the team.
  4. Write. The language wars, types, names and indentation.
  5. Review. What a second pair of eyes actually catches.
  6. Test. Which tests prevent the failures that matter.
  7. Ship. Fridays, outages and what continuous integration delivers.
  8. Age. Maintenance, technical debt, smells, refactoring and dependencies.

Three more parts step back: People (the 10x programmer and what actually helps developers), Machines (AI tools, version 1, the part most certain to age), and Evidence (how the field fools itself, and why nothing rates ~95%). The last part adds up what's left.

Each part follows the same pattern: the folklore, what the evidence shows, and why the surviving part works. Two boxes close each one. What doesn't work collects the claims that failed. Try this lists practices that follow from the evidence rather than from the enthusiasm.

A word of caution. This is a careful reading, not a meta-analysis. The ratings are estimates, the selection is one person's, and the research will move, fastest of all on AI. Some verdicts here will be wrong. The aim is not the final word, but an honest account of which numbers are load-bearing and which are leprechauns.

9 claims checked

The evidence behind this article

Every claim above, rated by the chance it holds up against the research. Open one for the sources and their exact words.

  1. Beller, Gousios, Panichella, Proksch, Amann, Zaidman (2019). Developer Testing in the IDE: Patterns, Beliefs, and Behavior · Study
  2. Beller, Gousios, Panichella, Zaidman (2015). When, how, and why developers (do not) test in their IDEs · Study
  3. Boehm, Basili (2001). Top 10 list [software development] · Source
  4. Brooks (1987). No Silver Bullet Essence and Accidents of Software Engineering · Source
  5. Buehler, Griffin, Ross (1994). Exploring the "planning fallacy": Why people underestimate their task completion times. · Study
  6. Dickey (1981). Programmer variability · Source
  7. Haskins, Stecklein, Dick, Moroney, Lovell, Dabney (2004). 8.4.2 Error Cost Escalation Through the Project Life Cycle · Source
  8. Joel, Rush, Barnes, David (2025). Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity · Study
  9. Jørgensen, Moløkken-Østvold (2006). How large are software cost overruns? A review of the 1994 CHAOS report · Study
  10. Lientz, Swanson, Tompkins (1978). Characteristics of application software maintenance · Study
  11. Menzies, Nichols, Shull, Layman (2016). Are delayed issues harder to resolve? Revisiting cost-to-fix of defects throughout the lifecycle · Study
  12. Molokken, Jorgensen (2003). A review of software surveys on software effort estimation · Meta-analysis
  13. Prechelt (2000). An empirical comparison of seven programming languages · Study
  14. Sackman, Erikson, Grant (1968). Exploratory experimental studies comparing online and offline programming performance · Study

Share

XLinkedIn