Software engineering, checked · Part 5 of 13
Write: the language wars
In 2014, Baishakhi Ray and colleagues did what the language wars had always lacked: they counted. They took 729 projects from GitHub, 80 million lines of code in 17 languages, and looked for a link between each project's language and its bugs. The answer came carefully worded. Language design had "a significant, but modest effect on software quality". Functional languages were somewhat better than procedural ones, and among the functional languages, static typing was somewhat better than dynamic. The authors added that these effects were "overwhelmingly dominated" by process factors such as project size, team size and commit size, and might reflect which kinds of people choose which languages. The caveats were in the abstract. What travelled was the headline: science had picked a side.
In 2019, a team led by Emery Berger repeated the analysis. They found mislabelled data and statistical errors. Of the 11 languages that had been associated with more or fewer defects, 4 remained, and for those "the effect size is exceedingly small". Their conclusion went further than the correction. Even with the flaws fixed, they wrote, too many sources of bias remain "to hope for a meaningful comparison of bug rates across languages" ~15 %.
This part is about writing code: the language it is in, and how it looks on the screen. The question is plain. Does either change how many bugs the code has, or how easily the next person understands it?
What types catch
The collapse of the GitHub result doesn't mean the language is irrelevant. It means whole-project bug counts are a blunt instrument.
A narrower question gets a clearer answer. Zheng Gao, Christian Bird and Earl Barr took 400 fixed bugs from 398 public JavaScript projects and asked, bug by bug, whether a type checker would have flagged it. Flow and TypeScript each caught about 15% ~85 %. The authors argue that this understates the benefit: these bugs had already survived testing and review, and a checker running on a developer's machine also catches the ones that never reach a commit.
Count at the level of whole projects, and the effect disappears. A comparison of 604 GitHub projects with 16 million lines of code found that TypeScript applications scored better on code quality and understandability, but were not less bug-prone, and their bugs took no less time to fix ~30 %. If anything, the TypeScript projects had the higher share of bug-fix commits. Across the studies, the pattern is the same: type checkers catch a real share of individual bugs, and project-level comparisons show at best a modest difference ~50 %. Both can be true: projects differ in a hundred ways besides their type system.
The functional languages fared worse. "Somewhat better", with the original authors' warning about personality types attached, was the most the data ever supported, and the reanalysis took most of that away ~30 %.
What about speed? Stefan Hanenberg's group ran a series of controlled experiments. With 49 students writing new code, a static type system made no difference to development time in either direction ~30 %. Where types did help was with code someone else wrote: maintenance tasks and unfamiliar APIs, where the types acted as documentation ~70 %. The samples are small groups of students and developers, and in one study with 27 subjects, types made some tasks faster and others slower. The direction is still worth keeping. Types are mostly for readers.
Memory safety is the exception
One language difference has a large, well-documented effect, and it is a narrow one. In big C and C++ products, about 70% of serious security vulnerabilities are memory-safety bugs. The figure comes from the vendors' own bug data, for Chrome, iOS, macOS and Android, and it is consistent across them ~85 %. It is not true of software in general: across all recorded CVEs, the share is about a third. For the large, long-lived C and C++ codebases much of the world runs on, it is the bulk of the problem.
This is a class of bug that a language can rule out by construction, and Rust tries to. An analysis of every memory-safety CVE in Rust up to 2020 found that all of them required unsafe code ~85 %. Rust keeps its promise, with a catch: unsafe code inside a library can leak a bug to the safe code that calls it. Still, the unsafe blocks are a far smaller surface to audit than a whole C++ codebase. And unlike "fewer bugs", "no memory-safety bugs outside unsafe code" is specific enough to check.
What doesn't change with the language
Some things stay put whatever you write in. An old rule of thumb, built into Boehm's COCOMO, says programmers write about the same number of lines per hour in any language. When Lutz Prechelt compared 80 implementations of the same program in seven languages, it held fairly well; only Java was clearly less productive than the scripting languages ~70 %. If lines per hour are roughly constant, a language that needs fewer lines for the job is faster to write in. In the same study, the scripts took less than half as long: a median of 3.1 hours against 10.0 ~50 %. It was one task, and the scripting times were self-reported.
Prechelt's more important finding sits in the same paper. The differences between programmers using the same language were typically larger than the differences between languages. Part 10 is about those differences. For this part, it is the thing to remember when the next language argument starts: who writes the code matters more than what they write it in.
Randomo
Syntax is where language arguments run hottest and the evidence is thinnest. Andreas Stefik and Susanna Siebert tried a direct test. They gave novices short programming tasks in several languages, one of which they had made up: Randomo, whose keywords were chosen at random. A placebo language. Novices using Perl and Java were no more accurate than novices using Randomo. Those using Quorum, Python and Ruby did better ~70 %.
Novices, short tasks: the result says nothing about experienced programmers. But it is a useful corrective. C-style syntax feels natural because we learnt it, not because it is.
Code is not prose to the brain
Programmers like to say that code is a language and reading it is like reading. The first fMRI study of code comprehension, with 17 participants, seemed to agree: it found five active brain regions, related to working memory, attention and language processing ~30 %. Later studies looked more closely. Anna Ivanova and colleagues first located each participant's own language network, then showed them code. In two experiments, the language network responded weakly or not at all. What responded strongly was the multiple-demand system, the brain's general-purpose network for hard problems. Another study, with 29 participants, found that the brain represents programming languages and natural languages distinctly.
The samples are small, as in most brain imaging, but the direction matters. If reading code is closer to solving a puzzle than to reading a sentence, then how code looks matters to the extent that it makes the puzzle easier.
Where does two to four spaces come from?
Indentation is the style question with the oldest evidence. In 1983, Miara and colleagues tested 86 subjects on Pascal programs indented by different amounts. The level of indentation had a significant effect on comprehension, and 2–4 spaces "had the highest mean score" ~15 %. Style guides have cited it ever since. In 2019, Bauer and colleagues repeated the experiment with eye tracking and found no indication that the indentation level affects comprehension or visual effort. Thirty-six years of citations, one replication, no effect.
Whether to indent at all is a different matter. In a 2023 randomised trial, participants needed 179% more time to answer questions about non-indented Java control flow than about indented code ~70 %. The effect is large when the nesting is tricky. How many spaces is a matter of taste; that you indent is not. Nothing in the evidence settles tabs versus spaces, which is probably for the best.
Names, case and atoms
Names are where code is most obviously people talking. Two developers naming the same thing almost never choose the same name: across 47 naming scenarios and 334 subjects, the median chance of a match was 6.9%. Yet most of the names chosen were understood by most other developers ~85 %. There is rarely a right name. There is a message to the next reader, which arrives or doesn't.
Full words help it arrive. 72 professional C# developers found defects 19% faster in code with full-word names than in code with letters and abbreviations ~70 %. The caveat is meaning: in another study, poor and misleading names helped no more than single letters. A long, wrong name is a single letter with extra typing.
The case style matters less than it feels. A 2009 study of 135 programmers and non-programmers found camelCase gave higher accuracy. An eye-tracking replication with 15 subjects, trained on underscores, found no difference in accuracy and faster recognition of under_score names ~30 %. Familiarity seems to matter more than the style. That is an argument for consistency within a codebase, not for either convention.
Then there are the atoms. Dan Gopstein and colleagues isolated tiny code patterns that make programmers misread code. In an experiment with 73 participants, the patterns significantly raised the rate of misunderstanding compared with equivalent code without them, and a second experiment found that removing them helps. The awkward detail is where they live: in large, popular projects, and in some style guides' recommendations ~85 %.
The metric that can't see
If understanding is what matters, it would help to measure it. Tools report complexity and readability scores as though they did. In 2017, Simone Scalabrino and colleagues tested them against 46 developers reading 50 code snippets. None of the metrics, existing or new, captured how understandable the code was, "not even the ones assumed to assess quality attributes strongly related with it, such as code readability and complexity" ~70 %. The verdict has softened since. A reanalysis found small but significant correlations, a validation study found that Cognitive Complexity tracks comprehension time and ratings, and an fMRI study added its weight to the doubts about the classic complexity metrics.
This is the gap between belief and measurement in miniature. The number on the dashboard is what gets measured. Whether the next developer understands the code is what matters, and the two are only loosely connected.
What doesn't work
- Choosing a language to cut bugs. On reanalysis, the famous GitHub result shrank to four languages and an exceedingly small effect ~15 %.
- Expecting TypeScript to lower the bug count. The checker catches real bugs; a 604-project comparison found TypeScript apps no less bug-prone ~30 %.
- Adopting static types to go faster. For writing new code, controlled experiments found no speed-up ~30 %.
- Enforcing 2–4 spaces as science. It rests on one 1983 Pascal study that didn't replicate ~15 %.
- Settling camelCase versus under_score on readability. Familiarity decides more than the style ~30 %.
Try this
- Add a type checker for the bugs it catches, not for the statistics. About one public JavaScript bug in seven was type-detectable, and the types document the code for whoever maintains it.
- Treat memory safety as the language decision with real stakes. For new security-sensitive code, a memory-safe language removes the largest class of vulnerabilities in C and C++. Keep unsafe blocks small and audit them, including the ones in your dependencies.
- Name for the reader. Full words, and names that mean what they say. A misleading name helps no more than a single letter.
- Write the plain version. Where a clever line and a plain one do the same thing, choose the plain one, whatever the style guide allows.
- Use complexity metrics as a prompt, not a gate. They can flag code worth a look; they can't tell you whether it is understandable. Ask someone who didn't write it.
20 claims checked
The evidence behind this article
Every claim above, rated by the chance it holds up against the research. Open one for the sources and their exact words.
- Avidan, Feitelson (2017). Effects of Variable Names on Comprehension: An Empirical Study · Study
- Barón, Wyrich, Wagner (2020). An Empirical Validation of Cognitive Complexity as a Measure of Source Code Understandability · Study
- Bauer, Siegmund, Peitek, Hofmeister, Apel (2019). Indentation: Simply a Matter of Style or Support for Program Comprehension? · Replication
- Berger, Hollenbeck, Maj, Vitek, Vitek (2019). On the Impact of Programming Languages on Code Quality · Replication
- Binkley, Davis, Lawrie, Morrell (2009). To camelcase or under_score · Study
- Bogner, Merkel (2022). To type or not to type? · Study
- Endrikat, Hanenberg, Robbes, Stefik (2014). How do API documentation and static typing affect API usability? · Study
- Feitelson, Mizrahi, Noy, Shabat, Eliyahu, Sheffer (2022). How Developers Choose Names · Study
- Floyd, Santander, Weimer (2017). Decoding the Representation of Code in the Brain: An fMRI Study of Code Review and Expertise · Study
- Gao, Bird, Barr (2017). To Type or Not to Type: Quantifying Detectable Bugs in JavaScript · Study
- Gopstein, Iannacone, Yan, DeLong, Zhuang, Yeh, Cappos (2017). Understanding misunderstandings in source code · Study
- Hanenberg (2010). An experiment about static and dynamic type systems · Study
- Haßler, Görz, Lipp (2025). A Comparative Study of Fuzzers and Static Analysis Tools for Finding Memory Unsafety in C and C++ · Study
- Hofmeister, Siegmund, Holt (2017). Shorter identifier names take longer to comprehend · Study
- Ivanova, Srikant, Sueoka, Kean, Dhamala, O'Reilly, Bers, Fedorenko (2020). Comprehension of computer code relies primarily on domain-general executive brain regions · Study
- Kleinschmager, Robbes, Stefik, Hanenberg, Tanter (2012). Do static type systems improve the maintainability of software systems? An empirical study · Study
- Mayer, Hanenberg, Robbes, Tanter, Stefik (2012). An empirical study of the influence of static type systems on the usability of undocumented software · Study
- Miara, Musselman, Navarro, Shneiderman (1983). Program indentation and comprehensibility · Study
- Morzeck, Hanenberg, Werger, Gruhn (2023). Indentation in Source Code: A Randomized Control Trial on the Readability of Control Flows in Java Code with Large Effects · Study
- Peitek, Apel, Parnin, Brechmann, Siegmund (2021). Program Comprehension and Code Complexity Metrics: An fMRI Study · Study
- Prechelt (2000). An empirical comparison of seven programming languages · Study
- Ray, Posnett, Filkov, Devanbu (2014). A large scale study of programming languages and code quality in github · Study
- Scalabrino, Bavota, Vendome, Linares-Vasquez, Poshyvanyk, Oliveto (2017). Automatically assessing code understandability: How far are we? · Study
- Sharif, Maletic (2010). An Eye Tracking Study on camelCase and under_score Identifier Styles · Study
- Siegmund, Kästner, Apel, Parnin, Bethmann, Leich, Saake, Brechmann (2014). Understanding understanding source code with functional magnetic resonance imaging · Study
- Stefik, Siebert (2013). An Empirical Investigation into Programming Language Syntax · Study
- Trockman, Cates, Mozina, Nguyen, Kästner, Vasilescu (2018). "Automatically assessing code understandability" reanalyzed · Replication
- Xu, Chen, Sun, Zhou, Lyu (2021). Memory-Safety Challenge Considered Solved? An In-Depth Study with All Rust CVEs · Study