Software Engineering, geprüft · Teil 13 von 13
What survives
Strip fifty years of software engineering folklore down to the claims that survive, and what's left is smaller than the conference talks promised and more useful than most of them delivered.
It isn't a methodology. Nothing in this series rated ~95%, and nobody who reads the evidence will come away believing in a process that fixes software. What remains is closer to a set of habits: at each step in the life of a change, a few things the evidence supports, and a longer list of things everyone repeats that it doesn't.
The life of a change on one page
- Decide. Assume the idea is a guess, and measure whether it worked. Best evidence: only about a third of the ideas tested at Microsoft moved the metric they were built for ~85 %.
- Estimate. Expect about 30% over, plan for the tail, and estimate from history, independently, with wider ranges. Best evidence: the 30% average across decades of surveys ~85 %; independent estimates combined beat most individuals ~85 %; 90% ranges catch the truth two times in three ~85 %.
- Organise. Make sure the people whose code depends on each other talk, and give code a clear owner. Best evidence: designs mirror organisations in most firms ~70 %; matching communication to dependencies speeds work ~70 %.
- Write. Pick types for the bugs they catch and memory safety for the vulnerabilities it removes; don't expect the language to fix quality. Best evidence: a type checker would catch about 15% of public JavaScript bugs ~85 %; most serious vulnerabilities in big C and C++ codebases are memory-safety bugs ~85 %.
- Review. Review to share understanding, and nudge the reviews that stall. Best evidence: three quarters of what review finds is maintainability, not bugs ~85 %; automatic reminders cut completion time sharply ~85 %.
- Test. Test the error handlers, fuzz the inputs. Best evidence: most catastrophic distributed failures came from errors the software had already detected ~70 %; random input still crashes standard tools ~85 %.
- Ship. Release often, fear the rush rather than the weekday, and make undo cheap. Best evidence: shorter release cycles didn't lower quality at Firefox ~70 %; most high-severity incidents ended without a code change ~70 %.
- Age. Watch where code changes, and keep dependencies moving. Best evidence: change history predicts defects better than code measures ~85 %; most developers don't update dependencies, even after an advisory ~85 %.
Read the evidence column again. The strongest entries are an experiment platform's base rate, decades of overrun surveys, a count of bugs in public repositories, and the incident logs of a large service. None of them is a methodology. All of them are measurements of what actually happened, which is what most of the folklore never was.
Three lessons from the checking
Follow the number home. The field's most famous numbers turned out to be origin stories. The 100x cost of a production bug leads back to unpublished 1970s data ~30 %. The 189% overrun came from executives asked for failure stories ~15 %. The 28:1 programmer came from a dozen people in 1968 ~15 %. "60–90% of cost is maintenance" came from a survey's rough estimates. Even the waterfall is a number of sorts, a diagram whose author drew it to say it rarely works ~15 %. In every case the figure travelled and the caveat stayed home. A useful reading habit follows: when a talk quotes a precise number about software, ask what was measured, in how many projects, and when. Usually nobody knows.
Software is people talking. The findings that held up best were rarely about code. Systems copy the organisations that build them. Code with a clear owner breaks less, and code changed near a reorganisation breaks more ~70 %. Review's real product is shared understanding ~70 %. A reminder bot speeds up pull requests more reliably than making them small. Being watched cut interview performance by more than half ~70 %. Configuration, where one team's assumptions meet another team's system, keeps turning up in outages. Brooks made this argument in 1975. His numbers aged badly; his point is the best-supported idea in the series.
What engineers believe is not what gets measured. Developers think they spend half their time testing; their IDEs recorded about a quarter ~30 %. Experienced developers felt 20% faster with AI tools while the stopwatch said 19% slower ~85 %. The same expert re-estimating the same task disagreed with themselves by 71% ~70 %. Features that senior stakeholders were excited about kept failing their experiments ~15 %. Developers said slow builds drove them from CI; their projects' behaviour said otherwise. This is the productivity series' last lesson again, in an engineering costume: the instrument most teams steer by, their own sense of how it's going, is the least reliable one they have.
The hall of shame
Some ideas are worth knowing mainly so you can stop repeating them:
- "A bug costs 100 times more in production." Old, unpublished data; 171 modern projects showed no consistent effect ~30 %.
- "Only 16% of projects succeed." A definition that even perfect forecasts fail ~15 %.
- "Iterative development was invented by agile." It goes back to the 1950s ~5 %.
- "Some people can't learn to program, and a test can tell." The camel test was retracted by its author ~5 %.
- "The language you choose decides how buggy your code is." The famous study shrank to almost nothing on reproduction ~15 %.
- "Formal models beat expert estimates." Experts won more often ~15 %.
- "Redundancy prevents outages." The failover chain is software too ~15 %.
- "Minor and patch releases don't break your code." In Maven, over a third of minor releases broke the API ~15 %.
None of these were fringe ideas. Each appears in textbooks, talks or team rules, and most were stated with a precise number. A reasonable test for the next piece of software folklore is whether it would make this list in five years.
Where to start
Don't adopt everything at once. The life of a change is a line, and a line is only as fast as its slowest step. Find the step where your team hurts most. If releases are frightening, it's Ship: make undo cheap before anything else. If everything is late, it's Estimate. If nobody knows who understands the billing code, it's Organise and Review. Fix that, watch the results for a quarter rather than a sprint, then look for the next one.
A minimal version fits on an index card:
- Before building, write down what the change should move, and check afterwards.
- Estimate from past work, independently, and widen the range.
- Give every part of the code an owner, and make sure the people who depend on each other talk.
- Review to understand, and chase the reviews that stall.
- Test what happens when things fail, not only when they work.
- Make every release easy to undo.
- Measure before you believe, especially about your tools.
A version number
Part 1 started with a leprechaun and a reminder that careful people get things wrong. This series will be wrong in places too. The ratings are judgements, the selection was one person's, and some findings, the AI studies above all, will look different within a year.
So treat this as version 1. Every claim here is rated so that it can be re-rated. When the evidence moves, the verdicts will move with it, and the short list will be rebuilt from whatever is still standing. That, after all, is the third lesson applied to the series itself: trust the measurement, not the feeling that it's finished.
34 Behauptungen geprüft
Die Belege hinter diesem Artikel
Jede Behauptung oben, bewertet nach der Chance, dass sie der Forschung standhält. Öffne eine für die Quellen und ihren genauen Wortlaut.
- Bacchelli, Bird (2013). Expectations, outcomes, and challenges of modern code review · Studie
- Behroozi, Parnin, Barik (2019). Hiring is Broken: What Do Developers Say About Technical Interviews? · Studie
- Behroozi, Shirolkar, Barik, Parnin (2020). Does stress impact technical interview performance? · Studie
- Beller, Bacchelli, Zaidman, Juergens (2014). Modern code reviews in open-source projects: which problems do they fix? · Studie
- Beller, Gousios, Panichella, Proksch, Amann, Zaidman (2019). Developer Testing in the IDE: Patterns, Beliefs, and Behavior · Studie
- Beller, Gousios, Panichella, Zaidman (2015). When, how, and why developers (do not) test in their IDEs · Studie
- Berger, Hollenbeck, Maj, Vitek, Vitek (2019). On the Impact of Programming Languages on Code Quality · Replikation
- Boehm, Basili (2001). Top 10 list [software development] · Quelle
- Cataldo, Herbsleb (2013). Coordination Breakdowns and Their Impact on Development Productivity and Software Failures · Studie
- Cataldo, Wagstrom, Herbsleb, Carley (2006). Identification of coordination requirements · Studie
- Colfer, Baldwin (2016). The mirroring hypothesis: theory, evidence, and exceptions · Quelle
- Dickey (1981). Programmer variability · Quelle
- Eveleens, Verhoef (2010). The rise and fall of the Chaos report figures · Studie
- Flyvbjerg, Budzier (2013). Why Your IT Project Might Be Riskier Than You Think · Studie
- Foucault, Palyart, Blanc, Murphy, Falleri (2015). Impact of developer turnover on quality in open-source software · Studie
- Gao, Bird, Barr (2017). To Type or Not to Type: Quantifying Detectable Bugs in JavaScript · Studie
- Ghosh, Shetty, Bansal, Nath (2022). How to fight production incidents? · Studie
- Graves, Karr, Marron, Siy (2000). Predicting fault incidence using software change history · Studie
- Grimstad, Jørgensen (2007). Inconsistency of expert judgment-based estimates of software development effort · Studie
- Gunawi, Hao, Suminto, Laksono, Satria, Adityatama, Eliazar (2016). Why Does the Cloud Stop Computing? · Studie
- Haskins, Stecklein, Dick, Moroney, Lovell, Dabney (2004). 8.4.2 Error Cost Escalation Through the Project Life Cycle · Quelle
- Hassan (2009). Predicting faults using the complexity of code changes · Studie
- Haßler, Görz, Lipp (2025). A Comparative Study of Fuzzers and Static Analysis Tools for Finding Memory Unsafety in C and C++ · Studie
- Joel, Rush, Barnes, David (2025). Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity · Studie
- Jorgensen (2014). What We Do and Don't Know about Software Development Effort Estimation · Studie
- Jørgensen (2004). A review of studies on expert estimation of software development effort · Metaanalyse
- Jørgensen (2007). Forecasting of software development work effort: Evidence on expert judgement and formal models · Metaanalyse
- Jørgensen, Moløkken (2004). Eliminating Over-Confidence in Software Development Effort Estimates · Studie
- Jørgensen, Moløkken-Østvold (2006). How large are software cost overruns? A review of the 1994 CHAOS report · Studie
- Khomh, Adams, Dhaliwal, Zou (2014). Understanding the impact of rapid releases on software quality · Studie
- Khomh, Dhaliwal, Zou, Adams (2012). Do faster releases improve software quality? An empirical case study of Mozilla Firefox · Studie
- Kohavi, Deng, Frasca, Walker, Xu, Pohlmann (2013). Online controlled experiments at large scale · Studie
- Kohavi, Longbotham, Sommerfield, Henne (2008). Controlled experiments on the web: survey and practical guide · Studie
- Kula, German, Ouni, Ishio, Inoue (2017). Do developers update their library dependencies? · Studie
- Kwan, Schroter, Damian (2011). Does Socio-Technical Congruence Have an Effect on Software Build Success? A Study of Coordination in a Software Project · Studie
- Larman, Basili (2003). Iterative and incremental developments. a brief history · Quelle
- Lung, Aranda, Easterbrook, Wilson (2008). On the difficulty of replicating human subjects studies in software engineering · Replikation
- MacCormack, Baldwin, Rusnak (2012). Exploring the duality between product and organizational architectures: A test of the “mirroring” hypothesis · Studie
- Maddila, Upadrasta, Bansal, Nagappan, Gousios, van Deursen (2023). Nudge: Accelerating Overdue Pull Requests toward Completion · Studie
- Mantyla, Lassenius (2009). What Types of Defects Are Really Discovered in Code Reviews? · Studie
- Menzies, Nichols, Shull, Layman (2016). Are delayed issues harder to resolve? Revisiting cost-to-fix of defects throughout the lifecycle · Studie
- Miller, Fredriksen, So (1990). An empirical study of the reliability of UNIX utilities · Studie
- Miller, Zhang, Heymann (2022). The Relevance of Classic Fuzz Testing: Have We Solved This One? · Replikation
- Mockus (2010). Organizational volatility and its effects on software defects · Studie
- Molokken, Jorgensen (2003). A review of software surveys on software effort estimation · Metaanalyse
- Prechelt (2000). An empirical comparison of seven programming languages · Studie
- Raemaekers, van Deursen, Visser (2017). Semantic versioning and impact of breaking changes in the Maven repository · Studie
- Rahman, Devanbu (2013). How, and why, process metrics are better · Studie
- Ray, Posnett, Filkov, Devanbu (2014). A large scale study of programming languages and code quality in github · Studie
- Rigby, Bird (2013). Convergent contemporary software peer review practices · Studie
- Sackman, Erikson, Grant (1968). Exploratory experimental studies comparing online and offline programming performance · Studie
- Sadowski, Söderberg, Church, Sipko, Bacchelli (2018). Modern code review · Studie
- Thongtanunam, McIntosh, Hassan, Iida (2015). Investigating Code Review Practices in Defective Files: An Empirical Study of the Qt System · Studie
- Yuan, Luo, Zhuang, Rodrigues, Zhao, Zhang, Jain, Stumm (2014). Simple Testing Can Prevent Most Critical Failures: An Analysis of Production Failures in Distributed Data-Intensive Systems · Studie
- Zimmermann, Staicu, Tenny, Pradel (2019). Small World with High Risks: A Study of Security Threats in the npm Ecosystem · Studie