Software Engineering, geprüft · Teil 12 von 13
Evidence: how the field fools itself
For decades, researchers have tried to predict which parts of a codebase will contain bugs. The recipe is familiar: train a machine-learning model on code metrics and change history, point it at a project, report how well it found the faulty modules. Paper after paper proposes a technique and reports that it beats the others.
In 2014, Martin Shepperd and colleagues collected 600 of those published results and asked what actually explained the differences in performance. The classifier, the thing each paper was about, explained 1.3%. The largest factor, at 31%, was the research group that ran the study ~15 %. Their summary was blunt: "It matters more who does the work than what is done."
Two years later, Chakkrit Tantithamthavorn and colleagues replied that "research group" was standing in for something more ordinary. Groups tend to reuse their own datasets and metrics, and once those were untangled, the choice of metrics mattered more than the group ~50 %. That reading is kinder to the researchers. It is not much kinder to the field. Either way, the published numbers were driven mostly by choices the papers weren't about.
Part 1 reported that none of the series' 283 new claims reached ~95%. This part explains why. The question is plain: why is software engineering's evidence so rarely strong enough to build on, and what can you do with it anyway?
It matters who does the work
Defect prediction is not unusual. Effort estimation has spent forty years comparing models, and the winners keep changing. Magne Jørgensen's summary of that literature: "Numerous studies compare the accuracy of estimation models and methods, with a large variety in who wins the estimation accuracy contest" ~85 %. A 2012 re-examination of published evaluations of estimation models, led by Shepperd, found the original conclusions unsafe. Measured against random guessing, even the strongest results had no more than a medium effect ~30 %.
In both fields, the sophisticated method keeps failing to beat the plain one. In a 2012 review, the defect models that performed well tended to use simple techniques such as Naive Bayes and logistic regression. That is a useful result in its own right. It also means that a paper announcing a new technique that wins by a few points is mostly reporting on its own setup: the data it chose, the metrics, the way it tuned things. The method is the headline; the setup is the result.
Methods that don't travel
Defect prediction has a second problem. A model that works on the project it was trained on often fails on the next one. In 2009, Thomas Zimmermann and colleagues ran 622 cross-project predictions, and most of them failed. Even using models from projects in the same domain, or with the same process, "does not lead to accurate predictions". Nine years later, a benchmark of 24 newer approaches by Steffen Herbold and colleagues concluded that cross-project prediction "still has not reached a point where the performance of the results is sufficient for the application in practice" ~30 %. The approach that came out best was based on standardising the data.
Estimation has the same lesson, learnt earlier. In 1987, Chris Kemerer tested published cost models on 15 business projects, and the models not built for that kind of work showed a significant need for calibration. Jørgensen's conclusion, decades later, is that companies "should try to build their own estimation models" ~85 %.
Sometimes the same question gets opposite answers in different places. Norman Fenton and Niclas Ohlsson famously found that the modules with the most faults before release were among those with the fewest after it. Two later replications, on other systems, found the reverse: faulty modules stayed faulty ~50 %. The same question, asked of different systems, got different answers.
A plausible reason runs through the whole series. What differs most between two projects is not the code but the people: who works on it, how they talk, how the organisation is cut. Software is people talking, and a model trained on one conversation transfers poorly to another.
Evaluations that can't be trusted
Some evaluations fail before the question of transfer even comes up. In 2018, Klees and colleagues assessed the experimental evaluations in 32 fuzzing papers. They found problems in every one, and showed that those problems "can indeed translate to actual wrong or misleading assessments" ~85 %. Fuzzing is one of the most practical areas of testing research, and its own evidence base needed testing.
AI coding benchmarks have the same disease, plus a new one: the model may have seen the exam. Several models did markedly worse on contest problems published after their training cutoff than on problems from before it. On SWE-bench Verified, models could name the buggy file from the issue text alone up to 76% of the time, but only up to 53% on repositories outside the benchmark ~85 %. That looks like memory, not reasoning.
The tests that grade the answers are often too weak to tell right from wrong. When researchers gave HumanEval about 80 times more tests, pass rates across 26 models fell by up to 29%, and the rankings changed. On SWE-bench Verified, 29.6% of the patches that passed still behaved differently from the developers' own fix ~85 %. Part 11 covers what this means for agents. The general point is older: a benchmark measures how well something does on the benchmark.
Replications that shrink
In 2018, a deep-learning model called Deep-SE was trained to estimate agile story points from issue text. On 23,313 issues from 16 open-source projects, it consistently outperformed three baselines: random guessing, the mean and the median. In 2022, Vali Tawosi and colleagues replicated it closely, on the original data plus 31,960 more issues. Deep-SE beat the median baseline, which simply estimates every issue at the median, with statistical significance in only 8 of 42 cases ~15 %.
The pattern recurs across the series. Part 5's language study, once its data was corrected, left four languages with an exceedingly small effect ~15 %. Part 6's link between review coverage and defects turned unstable in an exact replication, where models without the review measures fit as well ~50 %. In each case the first result was real enough to publish and too fragile to build on.
The striking result gets published, cited and turned into advice. The replication comes years later, if at all. Replications are rare in this field, and that, more than anything, is why none of these claims reaches ~95%.
Where do the big returns come from?
Some of the field's most quoted numbers come from the people selling the answer. Take process improvement. A 1997 paper on how the Capability Maturity Model helped Motorola reported a 5-to-1 return on investment at Tinker Air Force Base and 7.7 to 1 at Raytheon, along with a doubling of Raytheon's productivity. Motorola's own data showed quality improving by a factor of about two per maturity level. These are organisations reporting on programmes they championed, and no independent test of the ratio exists. The same paper has a quieter line: projects saw "an unexpected decrease in productivity" moving from level 2 to level 3 ~50 %.
The more careful version of the claim holds up better. In 30 products at one firm, higher maturity meant better quality and more development effort, but the time saved by fewer defects outweighed the extra effort ~70 %. Quality was roughly free there. One company is not an industry.
The modern equivalent is code-quality data from tool vendors. Across 39 proprietary codebases and 30,737 files, low-quality code had 15 times more defects than high-quality code, and its issues took 124% longer to resolve. The quality was measured with the vendor's own metric, on the vendor's own customers ~70 %. To their credit, the authors raised both problems themselves: the customers may not be representative, and causality may run the other way, with many defects producing low-quality code. The same paper estimated that technical debt wastes "up to 42%" of developers' time. That figure now travels beside the better-grounded 23%, which comes from 43 developers logging their own wasted time ~70 %.
None of this makes vendor data wrong. The direction is credible; the multiples are not constants.
Asking people instead of watching them
The cheapest evidence is a survey, and much of the field runs on it. DevOps is a good example. A 2019 survey of DevOps notes that the movement still lacks an agreed definition, and cites an earlier review that found academic studies of DevOps generally of low quality, with few studies showing it works. A paper on release engineering found empirical validation of its best practices "largely missing" ~30 %. The famous DORA finding, that speed and stability rise together, rests mainly on self-reported surveys, and the four key metrics are often measured by hand or by survey. Industrial data from Facebook supports the weaker version: critical issues stayed almost constant however often they deployed ~70 %.
Surveys and behaviour can disagree directly. In a replication of continuous-integration pain points, Widder and colleagues found strong evidence in their survey that people abandon CI because of long builds. In the data on 6,239 projects, very long builds did make leaving more likely, but explained the least of anything in the model, and the duration of the last build pointed the other way ~50 %. What people say drives them away and what does drive them away are different questions.
A survey answer is real data, but it is data about belief. Part 1's example still stands: developers think they test half their time, and measured, it was a quarter.
What doesn't work
- Picking a defect-prediction classifier. The classifier explained 1.3% of published performance ~15 %.
- Borrowing another project's defect model. Most cross-project predictions failed, and newer approaches still aren't good enough for practice ~30 %.
- Deep learning for story points. It rarely beat estimating the median every time ~15 %.
- Treating DevOps as empirically settled. The research base is thin, self-reported and short on definitions ~30 %.
- Quoting 5:1 returns. They come from the programme's champions, not from independent tests ~50 %.
Try this
- Ask who measured it, on whom. One company, students, open-source repositories, a survey or a vendor's customers: each limits what a number can tell you. Check before you repeat it.
- Trust direction over size. Many findings point the right way and get the multiple wrong. "Low-quality code has more defects" is a safer bet than "15 times".
- Prefer the simple baseline. Before adopting a clever model for estimates or defect hotspots, compare it with the median, last quarter's numbers or plain change history. The baseline often wins.
- Calibrate on your own data. Models and benchmarks built elsewhere transfer badly. Run the tool on your own repository, with your own history, before trusting it.
- Measure behaviour, not opinion. If the team says slow builds hurt, time the builds and watch what people do. The survey tells you what to look at; the measurement tells you what is true.
20 Behauptungen geprüft
Die Belege hinter diesem Artikel
Jede Behauptung oben, bewertet nach der Chance, dass sie der Forschung standhält. Öffne eine für die Quellen und ihren genauen Wortlaut.
- Adams, McIntosh (2016). Modern Release Engineering in a Nutshell -- Why Researchers Should Care · Quelle
- Andersson, Runeson (2007). A Replicated Quantitative Analysis of Fault Distributions in Complex Software Systems · Replikation
- Berger, Hollenbeck, Maj, Vitek, Vitek (2019). On the Impact of Programming Languages on Code Quality · Replikation
- Besker, Martini, Bosch (2018). Technical debt cripples software developer productivity · Studie
- Besker, Martini, Bosch (2019). Software developer productivity loss due to technical debt—A replication and extension study examining developers’ development work · Replikation
- Choetkiertikul, Dam, Tran, Pham, Ghose, Menzies (2018). A Deep Learning Model for Estimating Story Points · Studie
- Diaz, Sligo (1997). How software process improvement helped Motorola · Studie
- Fenton, Ohlsson (2000). Quantitative analysis of faults and failures in a complex software system · Studie
- Galinac Grbac, Runeson, Huljenic (2013). A Second Replicated Quantitative Analysis of Fault Distributions in Complex Software Systems · Replikation
- Hall, Beecham, Bowes, Gray, Counsell (2012). A Systematic Literature Review on Fault Prediction Performance in Software Engineering · Metaanalyse
- Harter, Krishnan, Slaughter (2000). Effects of Process Maturity on Quality, Cycle Time, and Effort in Software Product Development · Studie
- Herbold, Trautsch, Grabowski (2018). A Comparative Study to Benchmark Cross-Project Defect Prediction Approaches · Studie
- Jain, Han, Alex, Li, Yan, Zhang, Wang, Solar-Lezama, Sen, Stoica (2024). LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code · Studie
- Jorgensen (2014). What We Do and Don't Know about Software Development Effort Estimation · Studie
- Kemerer (1987). An empirical validation of software cost estimation models · Studie
- Kitchenham, Mendes, Travassos (2007). Cross versus Within-Company Cost Estimation Studies: A Systematic Review · Metaanalyse
- Klees, Ruef, Cooper, Wei, Hicks (2018). Evaluating Fuzz Testing · Studie
- Krutauz, Dey, Rigby, Mockus (2020). Do code review measures explain the incidence of post-release defects? Case study replications and bayesian networks · Replikation
- Leite, Rocha, Kon, Milojicic, Meirelles (2019). A Survey of DevOps Concepts and Challenges · Metaanalyse
- Liang, Garg, Moghaddam (2026). The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason · Studie
- Liu, Xia, Wang, Zhang (2023). Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation · Studie
- McIntosh, Kamei, Adams, Hassan (2014). The impact of code review coverage and code review participation on software quality: a case study of the qt, VTK, and ITK projects · Studie
- McIntosh, Kamei, Adams, Hassan (2015). An empirical study of the impact of modern code review practices on software quality · Studie
- Ray, Posnett, Filkov, Devanbu (2014). A large scale study of programming languages and code quality in github · Studie
- Rossi, Shibley, Su, Beck, Savor, Stumm (2016). Continuous deployment of mobile software at facebook (showcase) · Studie
- Sallin, Kropp, Anslow, Quilty, Meier (2021). Measuring Software Delivery Performance Using the Four Key Metrics of DevOps · Studie
- Savor, Douglas, Gentili, Williams, Beck, Stumm (2016). Continuous deployment at Facebook and OANDA · Studie
- Shepperd, Bowes, Hall (2014). Researcher Bias: The Use of Machine Learning in Software Defect Prediction · Metaanalyse
- Shepperd, MacDonell (2012). Evaluating prediction systems in software project estimation · Studie
- Tantithamthavorn, McIntosh, Hassan, Matsumoto (2016). Comments on “Researcher Bias: The Use of Machine Learning in Software Defect Prediction” · Studie
- Tawosi, Moussa, Sarro (2022). Agile Effort Estimation: Have We Solved the Problem Yet? Insights From a Replication Study · Replikation
- Thongtanunam, McIntosh, Hassan, Iida (2015). Investigating Code Review Practices in Defective Files: An Empirical Study of the Qt System · Studie
- Tornhill, Borg (2022). Code red: the business impact of code quality - a quantitative study of 39 proprietary production codebases · Studie
- Wang, Pradel, Liu (2026). Are “Solved Issues” in SWE-bench Really Solved Correctly? An Empirical Study · Studie
- Widder, Hilton, Kästner, Vasilescu (2019). A conceptual replication of continuous integration pain points in the context of Travis CI · Replikation
- Zimmermann, Nagappan, Gall, Giger, Murphy (2009). Cross-project defect prediction · Studie