Skip to main content

Software engineering, checked · Part 3 of 13

Estimate: the arithmetic of being late

November 19, 2026·8 min read
19 claims checked

In 1994, a consultancy called the Standish Group published a report with a memorable name and three memorable numbers. Only 16% of software projects succeeded. Nearly a third were cancelled. And the average project cost 189% of its original estimate. The CHAOS report became the most-quoted statistic in the industry. For twenty years it opened the sales pitch of every new method, tool and consultancy that promised to fix software development, which, by the numbers, badly needed fixing.

The numbers were wrong, and how they were wrong is more useful than the numbers ever were. This part is about the oldest complaint in software: it's late. The question is plain. Why are estimates so far off, and can they get better?

Where does 189% come from?

Start with the arithmetic. "189% of the original estimate" means an overrun of 89%, but the figure has been quoted as a 189% overrun so often that both versions circulate. Either way, it is far out of line. In 2006, Magne Jørgensen and Kjetil Moløkken-Østvold went back to the report. Standish had described its method: "We then called and mailed a number of confidential surveys to a random sample of top IT executives, asking them to share failure stories." Ask for failure stories and you get failure stories. Recalculated from Standish's own distribution of overruns, the average came out close to 89%, and every other survey from the period landed far lower ~15 %.

What do the other surveys say? Peer-reviewed studies from the 1980s onwards keep landing near the same figure: the typical software project overruns its estimate by about 30% ~85 %. Most projects overrun, somewhere between 60% and 80% of them ~85 %. Late is normal. Catastrophically late, on average, is not.

The 16% success rate has a different problem: the definition. Standish counted a project as a success only if it met its first estimate of cost, time and scope, and ignored projects that came in under. Chris Verhoef and Laurenz Eveleens applied those rules to an organisation whose forecasts were accurate and unbiased, as good as forecasting gets. It still scored only 59% "success" ~15 %. By Standish's definition, even perfect estimating fails four times in ten, because an unbiased estimate is too low about half the time. Later CHAOS reports showed the success rate climbing. That improvement, Jørgensen argues, mostly reflects better sampling, not better projects.

The average that doesn't exist

The 30% average is honest, but it hides the shape of the problem. When Bent Flyvbjerg and Alexander Budzier studied 1,471 IT projects, the average cost overrun was 27%. But one project in six was what they called a black swan, with an average cost overrun of 200% and a schedule overrun of almost 70% ~70 %.

A later study of 5,392 IT projects found why the averages mislead. Overruns follow a power law: many projects overrun a little, and a fat tail overruns enormously. In part of that tail the distribution is so heavy that, mathematically, the average overrun "does not exist": add more projects and the mean keeps moving instead of settling ~70 %.

The everyday version is the average wealth in a room, which changes completely when a billionaire walks in. For a project, the practical consequence is that "we'll be about 30% over" is the wrong kind of risk to plan for. The question that matters is whether this one is the billionaire.

The cone that wasn't

Barry Boehm drew a chart in 1981 that Steve McConnell later made famous as the cone of uncertainty. Early in a project, estimates might be off by a factor of four in either direction. As the project goes on, the range narrows, until at the end it closes on the truth. The cone is reassuring. Uncertainty is temporary, and the work itself will resolve it.

Todd Little checked it against real projects at Landmark Graphics. The uncertainty range was nearly identical from start to finish ~30 %. Eveleens and Verhoef found something similar in an organisation that padded its estimates: no cone at all. The cone shows what is possible when a team keeps re-estimating honestly. It doesn't happen on its own.

Inside the estimator

Most software estimates are expert judgement: somebody experienced looks at the work and names a number. The trouble starts there. People underestimate their own tasks, even familiar ones, because they plan from imagined scenarios rather than from past experience ~95 %.

Developers are also overconfident about their uncertainty. Asked for a range they are 90% sure contains the actual effort, they catch it only 60–70% of the time. In one industrial study, intervals labelled "almost certain" caught 35% ~85 %. And they are inconsistent: when seven experienced professionals re-estimated the same tasks months apart, their two estimates for a task differed by 71% on average ~70 %. That study is small; the direction is not in doubt.

Then there are anchors. Mention a number, any number, before people estimate, and their estimates drift towards it. In an experiment with 410 professional developers, an irrelevant anchor had a large effect, and a workshop on cognitive biases reduced it without removing it ~85 %. The client's budget, the manager's hope and last quarter's deadline are all anchors. Field experiments with real outsourcing companies found smaller effects than the lab, but not none.

Even the unit matters. In two small experiments, people asked to estimate in work-days produced much higher estimates than people asked in work-hours: 266 hours against 85 for the same task, among 22 project managers ~50 %. It's a tiny study and needs repeating. It is also exactly the kind of effect you'd never notice from the inside.

Models versus experts

If human judgement is this unreliable, the obvious fix is a formula. Decades of research have produced estimation models, from Boehm's COCOMO in 1981 to machine learning. They haven't won. In reviews of studies comparing the two, expert estimates were more accurate than the models in ten of sixteen ~15 %. Models built elsewhere need calibrating to your own data before they help, and sophisticated models rarely beat simple ones.

Neither side has improved much. Surveys from the 1980s and from recent years report similar average overruns, despite forty years of estimation research ~70 %.

What actually helps

The interventions with the best evidence are unglamorous.

Use history. Estimating from how long similar past work actually took, rather than imagining how this work will go, corrects much of the optimism. This is Kahneman's outside view, and Flyvbjerg's reference class forecasting ~70 %. In software, historical data and estimation checklists are among the best-documented ways to improve accuracy ~70 %.

Combine independent estimates. The average of several estimates from different people or methods is likely to be more accurate than most of the individual estimates, as long as they were made independently ~85 %. That condition is why planning poker exists: everyone shows their card at once, so nobody anchors on the loudest voice. The evidence for planning poker itself is thin and mixed. In one field study the consensus estimates beat the mechanical average of the same people's estimates, but tasks estimated individually were just as accurate ~50 %.

Watch how the work is bought. In competitive bidding, a client who picks the lowest price is likely to pick the most optimistic bidder, and with them the overrun ~70 %. Underestimation is concentrated in exactly these situations. The winner's curse isn't a software problem, but software has it badly.

What doesn't work

  • Quoting the CHAOS report. The 189% came from solicited failure stories, and the 16% from a definition that even perfect forecasts fail ~15 %.
  • Waiting for the cone to close. Uncertainty doesn't shrink on its own; in real project data it often stayed flat ~30 %.
  • Trusting a model over the people. Off-the-shelf models lose to experts more often than they win ~15 %.
  • Planning for the average overrun. In a fat tail, the average is the wrong number; one project in six blows up ~70 %.
  • A single 90% range from a single person. It will be too narrow, and next month the same person would say something different.

Try this

  • Estimate from your own history. Before committing to a number, look up how long the last few similar pieces of work actually took.
  • Collect estimates before discussing them. Ask several people independently, then average or discuss the spread. Don't mention the budget, the deadline or anyone's guess first.
  • Widen your ranges. If your 90% intervals catch the truth two times in three, they are about half as wide as they should be.
  • Plan for the tail. Decide in advance what happens if this is the one in six: a stop-loss, a scope cut, a point at which you re-plan rather than push on.
  • Be wary of the lowest bid. If you buy software, the cheapest credible offer is also the one most likely to be wrong.

19 claims checked

The evidence behind this article

Every claim above, rated by the chance it holds up against the research. Open one for the sources and their exact words.

  1. Buehler, Griffin, Ross (1994). Exploring the "planning fallacy": Why people underestimate their task completion times. · Study
  2. Eveleens, Verhoef (2010). The rise and fall of the Chaos report figures · Study
  3. Flyvbjerg (2007). Curbing Optimism Bias and Strategic Misrepresentation in Planning: Reference Class Forecasting in Practice · Study
  4. Flyvbjerg, Budzier (2013). Why Your IT Project Might Be Riskier Than You Think · Study
  5. Flyvbjerg, Budzier, Lee, Keil, Lunn, Bester (2022). The Empirical Reality of IT Project Cost Overruns: Discovering A Power-Law Distribution · Study
  6. Grimstad, Jørgensen (2007). Inconsistency of expert judgment-based estimates of software development effort · Study
  7. Jorgensen (2014). What We Do and Don't Know about Software Development Effort Estimation · Study
  8. Jorgensen (2015). The effect of the time unit on software development effort estimates · Study
  9. Jørgensen (2004). A review of studies on expert estimation of software development effort · Meta-analysis
  10. Jørgensen (2007). Forecasting of software development work effort: Evidence on expert judgement and formal models · Meta-analysis
  11. Jorgensen, Grimstad (2011). The Impact of Irrelevant and Misleading Information on Software Development Effort Estimates: A Randomized Controlled Field Experiment · Study
  12. Jørgensen, Moløkken (2004). Eliminating Over-Confidence in Software Development Effort Estimates · Study
  13. Jørgensen, Moløkken-Østvold (2006). How large are software cost overruns? A review of the 1994 CHAOS report · Study
  14. Little (2006). Schedule estimation and uncertainty surrounding the cone of uncertainty · Study
  15. Molokken-Ostvold, Haugen (2007). Combining Estimates with Planning Poker--An Empirical Study · Study
  16. Molokken, Jorgensen (2003). A review of software surveys on software effort estimation · Meta-analysis
  17. Shepperd, Mair, Jørgensen (2018). An experimental evaluation of a de-biasing intervention for professional software developers · Study

Share

XLinkedIn