Software engineering, checked · Part 6 of 13
Review: what other eyes catch
Ask developers why they review code and most give the same answer: to find bugs. In 2013, Alberto Bacchelli and Christian Bird put the question to developers at Microsoft and heard exactly that. Finding defects was their top motivation for reviewing, and the outcome they expected.
Then the researchers read the review comments and sorted them into nine categories. Defects came fourth. Of the comments they classified, 78, about 14%, were about defects ~30 %. The rest were mostly about making the code better: clearer, more consistent, easier to change. A dozen pointed the author to resources elsewhere. Reviews, the authors concluded, were less about defects than expected and more about knowledge transfer and team awareness.
Pull requests have made review a daily ritual for many teams, and it isn't free: someone waits, someone else stops to read. The question is plain. What does a second pair of eyes actually catch, and is that worth what it costs?
What review finds
Mika Mäntylä and Casper Lassenius classified what reviews turned up and found that 75% of the defects found during review do not affect the visible functionality of the software. Moritz Beller and colleagues repeated the exercise on open-source projects and found what they called a strikingly similar 75:25 ratio of maintainability problems to functional ones. In the Qt project, the concerns raised in review tended to improve evolvability, whether the file later turned out defective or not ~85 %.
That isn't review failing. A reader who isn't the author is good at noticing the misleading name, the duplicated logic, the function that does three things. Functional bugs are harder to see by reading. A second Microsoft team, Jacek Czerwonka and colleagues, gave their 2015 paper a blunt title, "Code Reviews Do Not Find Bugs", and argued that reviews often miss the functional problems that should block a change. Google's account of its process is more diplomatic: "Defect finding is welcomed but not the only focus."
What developers believe review is for and what the comments show it doing are two different things. The practice has quietly been doing the second while being justified by the first.
Reviewed code still breaks
If review were a bug filter, approved code would be clean. At Mozilla, Oleksii Kononenko and colleagues traced reviewed changes forward and found that 54% of them introduced bugs ~15 %. An approval says someone read the change. It says little about whether the change is correct.
Does more review at least mean fewer defects? Shane McIntosh and colleagues studied Qt, VTK and ITK and estimated that components with low review coverage and participation would have up to two and five additional post-release defects respectively. In Qt, files that later turned out defective had been reviewed less rigorously than clean ones. But when Andrey Krutauz and colleagues replicated the study on the original Qt data and on Chrome, the influence of the review measures was highly unstable, and models without any review predictors fitted as well or better ~50 %. Prior defects, size and who wrote the code predict far more. Review probably helps quality. How much is not pinned down.
Review is people talking
The payoff that shows up most clearly is knowledge. Peter Rigby and Christian Bird measured what developers know about a codebase by the files they have seen, and found that peer review increased the number of distinct files a developer knows about by 66% to 150%, depending on the project ~70 %. Files seen is a proxy, not knowledge tested, but the mechanism is plain: every review is one more person who has read that part of the system.
Google built its process around this. Caitlin Sadowski and colleagues describe review there as introduced for readability and education, and their data shows how lightweight it became. Across 9 million reviewed changes, the median change had one reviewer, touched 24 lines and was fully reviewed in under four hours, against medians of about 15 to 20 hours reported at AMD, Chrome OS and Microsoft ~85 %. One company, not a law, but it shows that a review whose main job is a conversation can be quick.
Conversations carry the biases of the people in them. In a large analysis at Google, Emerson Murphy-Hill and colleagues found that White, male and younger engineers receive less pushback in code review than engineers in other demographic groups ~70 %. On GitHub, Josh Terrell and colleagues found that among contributors who were outsiders to a project, men's pull requests were accepted more often than women's when gender was identifiable. The evidence comes from few settings, but it is reason enough to look at who gets the long comment threads on your team.
The rituals that didn't hold
Modern review descends from something heavier. In the 1970s, Michael Fagan formalised inspections at IBM, with a moderator, preparation, a meeting, checklists, a log of defects. Fagan reported that inspection usually detects more defects, at lower cost, than machine testing, and a large study at Bell-Northern later confirmed inspection as one of the most efficient ways to remove defects ~70 %. "More than testing" goes too far: Barry Boehm and Victor Basili noted that reviews, analysis tools and testing catch different classes of defects. They complement each other.
Its parts fared worse when tested one at a time.
The meeting. Lawrence Votta's data from Bell Labs showed that inspection meetings are not as beneficial as managers and developers think they are, and that about 20% of the inspection interval was lost to scheduling them. Controlled experiments by Adam Porter and colleagues found the same: the meeting adds few defects that reviewers hadn't already found alone ~70 %. It may still filter false alarms, but it costs calendar time.
The checklist. Les Hatton tested checklists on 308 industrial inspections and found no evidence that they significantly improved them ~30 %. Who inspected mattered far more: individual performance varied by a factor of ten in faults found per unit of time.
The team size. In a randomised field experiment, Porter and colleagues found inspection teams of one, two or four reviewers equally effective, while some combinations dramatically lengthened the interval. Pooling independent reviewers does find more: in Hatton's data, two-person teams found 76% of the faults on average. And Google gets by with one ~50 %. The honest answer is that it depends.
Small changes, or a nudge?
"Keep your pull requests small" is the most common review advice there is, and half of it holds. Amiangshu Bosu and colleagues analysed 1.5 million review comments at Microsoft and found that the more files in a change, the lower the proportion of comments that are useful to its author. Larger changes also carry more vulnerabilities ~70 %. Reviewers do write more comments on big changes, up to a peak of 12.5 per change at around 1,250 lines in Google's data. A larger share of what they say is of no use to the author.
The other half is that small changes get merged faster. Gunnar Kudrjavets and colleagues checked 845,000 GitHub pull requests, then 401,790 reviews from Gerrit and Phabricator. Size and composition did not relate to time-to-merge ~30 %. Small changes do get first feedback sooner: at Google, under an hour for small changes and about five hours for very large ones. Getting them finished is another matter.
What did speed things up was social, not structural. Chandra Maddila and colleagues built a tool at Microsoft that sends a reminder when a pull request is overdue. In a randomised trial on 147 repositories, it cut resolution time by 60% across 8,500 pull requests, and the effect held when it was rolled out to 8,000 repositories. Developers rated 73% of the notifications positive ~85 %. One company again, but it suggests a pull request mostly waits because nobody has got round to it, not because it is big.
Where does "enough eyeballs" come from?
"Given enough eyeballs, all bugs are shallow" comes from Eric Raymond's 1997 essay The Cathedral and the Bazaar, which named it Linus's law after Linus Torvalds. It was an argument for open-source development, drawn from experience, not a measurement.
The research supports half of it. In one small study of hired developers reviewing a web application for security flaws, Anne Edmundson and colleagues estimated that ten independent reviewers would have about an 80% chance of finding all the vulnerabilities, and fifteen about 95%. More eyes reading the code do find more. But in the Red Hat Linux kernel, Andrew Meneely and Laurie Williams found that files changed by nine or more developers were 16 times more likely to have a vulnerability than files changed by fewer ~30 %. Many hands changing the code is a different thing from many eyes reading it, and it seems to make code less secure, not more ~70 %. That is one codebase, and a correlation rather than a cause. It is also a fair summary of the slogan's fate: it counted eyes, and the data counts hands.
The pair programming correction
If one reviewer after the fact catches a little, what about a second programmer throughout? Extreme Programming made pairing a core practice in the late 1990s, and the early research was enthusiastic. Laurie Williams and colleagues argued in 2000 that pairing yields better products in less time, with happier, more confident programmers, and the figure that travelled was that it costs only about 15% more effort than two people working alone.
The largest experiment, by Erik Arisholm and colleagues with 295 professionals, found no general gain in time or correctness, and an 84% increase in the effort needed to finish the tasks correctly ~30 %. A meta-analysis by Jo Hannay and colleagues found a small positive effect on quality, a medium positive effect on duration and a medium negative effect on effort, with signs of publication bias ~70 %. A 14-month industrial study found a perceivable but small drop in defects.
Nor does a pair beat its own best member. In a controlled experiment by VenuGopal Balijepally and colleagues, pairs performed at the level of the better programmer in the pair, not above it: no assembly bonus, which is the usual finding in small-group research ~30 %.
The payoff is real but narrow. On the more complex system in Arisholm's experiment, pairs had 48% more correct solutions, and the benefit went mainly to juniors. Hannay's analysis found the same shape: pairing is faster on simple tasks and better on complex ones, and the quality comes at a price of considerably greater effort ~70 %. Pairing is a tool for hard problems and for teaching, not a cheap, general quality upgrade.
What doesn't work
- Treating an approval as a bug filter. More than half of reviewed changes at Mozilla introduced bugs ~15 %.
- Inspection by checklist. The one formal test, on 308 industrial inspections, found no significant benefit ~30 %.
- Splitting pull requests to get them merged faster. Across more than a million reviews, size didn't predict time-to-merge ~30 %.
- Counting on many eyeballs. Files changed by many developers had more vulnerabilities, not fewer ~30 %.
- Pairing as a cheap quality fix. It costs far more than 15% extra effort, and pays off mainly on complex tasks ~30 %.
Try this
- Review for the next reader. Ask whether the reviewer could maintain this code, and rotate reviewers so more people know each part of the system. Don't count on it for functional bugs; tests and tools catch different ones.
- Keep changes small for the review's sake. Big changes get less useful comments; just don't expect small ones to merge faster on their own.
- Automate the reminder. At Microsoft, a nudge on overdue pull requests cut resolution time by 60%. No size rule has shown anything like it.
- Drop the ceremony that doesn't pay. No meeting by default, one reviewer for routine changes, more independent readers for security-critical code.
- Pair on purpose. Use it for complex work and for bringing juniors up to speed, not as a standing rule.
20 claims checked
The evidence behind this article
Every claim above, rated by the chance it holds up against the research. Open one for the sources and their exact words.
- Arisholm, Gallis, Dyba, Sjoberg (2007). Evaluating Pair Programming with Respect to System Complexity and Programmer Expertise · Study
- Bacchelli, Bird (2013). Expectations, outcomes, and challenges of modern code review · Study
- Balijepally, Mahapatra, Nerur, Price (2009). Are Two Heads Better than One for Software Development? The Productivity Paradox of Pair Programming1 · Study
- Beller, Bacchelli, Zaidman, Juergens (2014). Modern code reviews in open-source projects: which problems do they fix? · Study
- Boehm, Basili (2001). Top 10 list [software development] · Source
- Bosu, Carver, Hafiz, Hilley, Janni (2014). Identifying the characteristics of vulnerable code changes: an empirical study · Study
- Bosu, Greiler, Bird (2015). Characteristics of Useful Code Reviews: An Empirical Study at Microsoft · Study
- Czerwonka, Greiler, Tilford (2015). Code Reviews Do Not Find Bugs. How the Current Code Review Best Practice Slows Us Down · Study
- di Bella, Fronza, Phaphoom, Sillitti, Succi, Vlasenko (2013). Pair Programming and Software Defects--A Large, Industrial Case Study · Study
- Edmundson, Holtkamp, Rivera, Finifter, Mettler, Wagner (2013). An Empirical Study on the Effectiveness of Security Code Review · Study
- Fagan (1986). Advances in software inspections · Study
- Hannay, Dybå, Arisholm, Sjøberg (2009). The effectiveness of pair programming: A meta-analysis · Meta-analysis
- Hatton (2008). Testing the Value of Checklists in Code Inspections · Study
- Kononenko, Baysal, Guerrouj, Cao, Godfrey (2015). Investigating code review quality: Do people and participation matter? · Study
- Krutauz, Dey, Rigby, Mockus (2020). Do code review measures explain the incidence of post-release defects? Case study replications and bayesian networks · Replication
- Kudrjavets, Nagappan, Rastogi (2022). Do small code changes merge faster? a multi-language empirical investigation · Study
- Maddila, Upadrasta, Bansal, Nagappan, Gousios, van Deursen (2023). Nudge: Accelerating Overdue Pull Requests toward Completion · Study
- Mantyla, Lassenius (2009). What Types of Defects Are Really Discovered in Code Reviews? · Study
- McIntosh, Kamei, Adams, Hassan (2014). The impact of code review coverage and code review participation on software quality: a case study of the qt, VTK, and ITK projects · Study
- McIntosh, Kamei, Adams, Hassan (2015). An empirical study of the impact of modern code review practices on software quality · Study
- Meneely, Williams (2009). Secure open source collaboration · Study
- Murphy-Hill, Jaspan, Egelman, Cheng (2022). The pushback effects of race, ethnicity, gender, and age in code review · Study
- Nosek (1998). The case for collaborative programming · Study
- Porter, Johnson (1997). Assessing software review meetings: results of a comparative analysis of two experimental studies · Study
- Porter, Siy, Toman, Votta (1997). An experiment to assess the cost-benefits of code inspections in large scale software development · Study
- Rigby, Bird (2013). Convergent contemporary software peer review practices · Study
- Russell (1991). Experience with inspection in ultralarge-scale development · Study
- Sadowski, Söderberg, Church, Sipko, Bacchelli (2018). Modern code review · Study
- Terrell, Kofink, Middleton, Rainear, Murphy-Hill, Parnin, Stallings (2017). Gender differences and bias in open source: pull request acceptance of women versus men · Study
- Thongtanunam, McIntosh, Hassan, Iida (2015). Investigating Code Review Practices in Defective Files: An Empirical Study of the Qt System · Study
- Votta (1993). Does every inspection need a meeting? · Study
- Williams, Kessler, Cunningham, Jeffries (2000). Strengthening the case for pair programming · Study