Software Engineering, geprüft · Teil 8 von 13
Ship: Fridays, outages and the CI promise
Every team has the rule, even if nobody wrote it down. It's Friday afternoon, the change is ready, the tests are green, and someone says it: not today. Never deploy on a Friday. The reasoning is sound enough. Whoever deploys is tired, half the people who could help have gone home, and anything that breaks will break into the weekend.
There is even a study. In 2005, Jacek Śliwerski, Thomas Zimmermann and Andreas Zeller traced which changes in the Mozilla and Eclipse histories later needed a fix, and found the likelihood highest on Friday. In Eclipse, a Friday change needed a fix 12.2% of the time, against an 11.4% average ~30 %. That's less than one change in a hundred. When Eyolfson and colleagues repeated the question on the Linux kernel and PostgreSQL, the differences between days of the week were small and varied by project.
The Friday rule is about a fear bigger than Fridays: that every release is a risk, so fewer releases mean less risk. The question is plain. Does shipping more often make things break more?
What the Friday study measured
The study behind the rule didn't look at deployments at all. It looked at when a change was committed, in two open-source projects, and asked whether that change was later fixed. That's a reasonable proxy for a bug, and a long way from "a Friday release takes down production". The saying travelled from commits to deploys without anyone checking the second.
A better-supported version of the worry concerns rush, not the calendar. Seven years of data from Facebook's mobile releases showed that changes pushed on the day the release branch was cut were of lower quality on average ~70 %. That is one company, but it fits what anyone who has worked to a branch cut will recognise: the last-minute change, squeezed in because the next train leaves in two weeks. The risk sits with the deadline, not with Friday. If you only release on Thursdays, Thursday becomes the rushed day.
Which clock you measure
Continuous integration promises speed. Whether it delivers depends on which clock you look at.
Michael Hilton and colleagues studied 34,544 GitHub projects. Those using CI released 0.54 times a month, those without 0.24, and projects released more often after adopting it ~70 %. Bogdan Vasilescu and colleagues looked at 246 projects and found that teams with CI merged more outside contributions without any rise in user-reported bugs ~70 %. Later work paints a more nuanced picture: pull requests took longer to process, and the benefits depend on how well a team keeps up the practices behind the tool. All of it is open source and observational.
Now look at a single change. Bernardo and colleagues measured how long merged pull requests took to reach users in 87 projects before and after CI. Only 51.3% delivered them faster ~30 %. The main reason was success: after adopting CI, many more pull requests arrived, and the queue grew. A replication with a control group found the same. Hilton's data shows the median pull request merged 1.6 hours faster with CI, but merged is not shipped.
So CI doubles releases and leaves half of projects no quicker per change. Both are true. A team that adopts CI to get each fix to users sooner may be disappointed; a team that wants to absorb more work without losing quality has better evidence on its side.
There is a quieter effect too. Across 685 GitHub projects, pull requests drew about one fewer review comment after CI was introduced, for the same amount of rework ~70 %. The authors call CI a silent helper. Some of the conversation a reviewer would have had, "does this even build?", the machine now has instead.
CI theatre
Running a CI server is not the same as integrating continuously. Of 1,270 projects on Travis CI, about 60% committed infrequently, and 85% had left a build broken for more than four days ~70 %. The tool was there; the practice wasn't. The one rule most projects did keep was the cheap one: builds mostly ran in under ten minutes.
CI also needs something to run. Failing tests are the most common reason builds fail: in 2.6 million Java and Ruby builds on Travis CI, testing was the single most important cause ~85 %. That is the point. A build without tests checks that the code compiles, and little else. In David Widder and colleagues' replication of CI pain points, projects with more test files were less likely to abandon CI: each 1% increase in test files cut the chance of abandoning by 16% ~70 %.
What developers say about slow builds
Ask developers why they leave a CI service, and long builds come up again and again. Widder's team asked, and the survey said exactly that. Then they modelled what 6,239 projects actually did. Very long builds did make abandoning more likely, but build time explained the least of anything in the model, and the duration of the last build pointed the other way ~50 %.
This is the series' third thread in miniature. Slow builds are annoying, and annoyance is memorable. That doesn't make them the reason people leave. In the same data, projects with fewer tests were more likely to leave.
Fast and stable
If the Friday rule is right, releasing more often should mean more failures. The best industrial data says it doesn't.
When Firefox moved to six-week releases, Foutse Khomh and colleagues found users didn't see significantly more post-release bugs, and bugs were fixed faster ~70 %. There were costs: crashes came earlier in a session, and a follow-up found harder bugs being pushed into later releases. At Facebook and OANDA, the number of critical issues from deployments stayed almost constant however many deployments there were, while Facebook's engineering team grew twentyfold ~70 %. These are insider case studies, and Facebook did go through a period of more hotfixes. Still, across Firefox and Facebook the trade-off everyone expects doesn't show up: failures didn't rise with deployment frequency ~70 %.
The stronger version comes from Accelerate, the 2018 book by Nicole Forsgren, Jez Humble and Gene Kim: speed and stability don't just coexist, they rise together, and the four DORA metrics predict how well an organisation does. That stronger claim rests mainly on self-reported surveys. In practice, the four metrics are often measured by hand or by survey, with few data points. Two of the book's best-known claims couldn't be checked at all: that the metrics predict organisational performance, and the multiples by which "elite performers" out-deploy everyone else. Both sit in the book and in grey reports. No peer-reviewed critique of the survey method turned up either.
Step back, and the field's evidence base looks thin. Reviews find that DevOps has no agreed definition, that its academic studies are of low quality, and that empirical validation of release-engineering practices is largely missing ~30 %. Most of what exists is CI in open source. The practices may well be right. The evidence that they are is narrower than the conference talks suggest.
Where outages come from
The most-quoted number about outages is that 70% of them come from changes. It is usually traced to Google's Site Reliability Engineering book, and no outage study in the library measures it directly. What the studies do measure is less tidy. Haryadi Gunawi and colleagues read the public reports of 597 outages at 32 major cloud services. Among outages with a known cause, upgrades were the largest single one, at 16% ~50 %. In some cases the upgrade had been tested; it was pushing it to the whole system that broke things. At Microsoft Teams, Ghosh and colleagues found 40% of high-severity incidents came from code or configuration bugs, and 60% from infrastructure, deployment and service dependencies. Changes matter. "Most" is not measured.
Configuration is the other usual suspect. Systems researchers call misconfiguration one of the dominant causes of failure, and in outage studies it makes up about 10–13% of known root causes, behind upgrades, network failures and bugs ~70 %. Where configuration is written as code, wrong configuration values are the most common defect category, in OpenStack and in a 2025 replication on newer tools and proprietary projects ~85 %. Configuration is where one team's assumptions meet another team's system. It goes wrong the way conversations go wrong.
Redundancy doesn't save you either. Those 597 outages happened despite pervasive redundancy, because redundancy only works if the whole recovery chain works: complete detection, flawless failover code, working backups ~15 %. Each link is software, and each can fail. Detection is a weak link. At Microsoft Teams, monitors caught only about half of the high-severity incidents, often because a monitor existed but missed the problem ~30 %. Gunawi's reports show bugs in the monitoring systems themselves.
How do incidents end? At Microsoft Teams, more than 90% of high-severity incidents were mitigated without changing code: rollbacks, configuration or infrastructure changes, restarts ~70 %. That is one service, and permanent fixes often came later. But it says something about what makes recovery fast. It isn't heroic debugging at 2 AM on a Saturday. It's being able to undo.
What doesn't work
- Banning Friday deploys on the evidence. The study measured commits, not deploys, and found a gap of less than one point in one project ~30 %.
- Expecting CI to get each change to users sooner. Only about half of projects delivered faster after adopting it ~30 %.
- Relying on redundancy alone. Outages happen anyway when detection, failover or backups fail ~15 %.
- Trusting monitoring to find the incident. At Microsoft Teams it caught about half ~30 %.
- Calling DevOps practice proven. The independent evidence is thin, and the famous numbers come from surveys ~30 %.
Try this
- Fear the rush, not the weekday. Watch changes squeezed in just before a branch cut or deadline more closely than changes made on a Friday.
- Pick your clock before adopting CI. Measure the time from merge to users as well as release frequency, so you know which one you improved.
- Integrate, don't just install. Commit often, fix broken builds the same day, and invest in tests before tuning build speed.
- Make undo cheap. Since most incidents end with a rollback, a config change or a restart, rehearse those, and treat configuration changes like code changes.
- Test the recovery chain. Check that monitors fire and failover works, deliberately, before an outage checks it for you.
20 Behauptungen geprüft
Die Belege hinter diesem Artikel
Jede Behauptung oben, bewertet nach der Chance, dass sie der Forschung standhält. Öffne eine für die Quellen und ihren genauen Wortlaut.
- Adams, McIntosh (2016). Modern Release Engineering in a Nutshell -- Why Researchers Should Care · Quelle
- Beller, Gousios, Zaidman (2016). Oops, my tests broke the build: An analysis of Travis CI builds with GitHub · Studie
- Bernardo, da Costa, Kulesza (2018). Studying the impact of adopting continuous integration on the delivery time of pull requests · Studie
- Cassee, Vasilescu, Serebrenik (2020). The Silent Helper: The Impact of Continuous Integration on Code Reviews · Studie
- Eyolfson, Tan, Lam (2011). Do time of day and developer experience affect commit bugginess? · Studie
- Felidré, Furtado, Costa, Cartaxo, Pinto (2019). Continuous Integration Theater · Studie
- Ghosh, Shetty, Bansal, Nath (2022). How to fight production incidents? · Studie
- Gunawi, Hao, Suminto, Laksono, Satria, Adityatama, Eliazar (2016). Why Does the Cloud Stop Computing? · Studie
- Guo, Leitner (2019). Studying the impact of CI on pull request delivery time in open source projects—a conceptual replication · Replikation
- Hilton, Tunnell, Huang, Marinov, Dig (2016). Usage, costs, and benefits of continuous integration in open-source projects · Studie
- Khomh, Adams, Dhaliwal, Zou (2014). Understanding the impact of rapid releases on software quality · Studie
- Khomh, Dhaliwal, Zou, Adams (2012). Do faster releases improve software quality? An empirical case study of Mozilla Firefox · Studie
- Leite, Rocha, Kon, Milojicic, Meirelles (2019). A Survey of DevOps Concepts and Challenges · Metaanalyse
- Oliveira, Paiva, Pereira, Brunet (2025). A Defect Taxonomy for Infrastructure as Code: A Replication Study · Replikation
- Rahman, Farhana, Parnin, Williams (2020). Gang of eight · Studie
- Rausch, Hummer, Leitner, Schulte (2017). An Empirical Analysis of Build Failures in the Continuous Integration Workflows of Java-Based Open-Source Software · Studie
- Rossi, Shibley, Su, Beck, Savor, Stumm (2016). Continuous deployment of mobile software at facebook (showcase) · Studie
- Sallin, Kropp, Anslow, Quilty, Meier (2021). Measuring Software Delivery Performance Using the Four Key Metrics of DevOps · Studie
- Santos, Costa, Kulesza (2022). Investigating the Impact of Continuous Integration Practices on the Productivity and Quality of Open-Source Projects · Studie
- Savor, Douglas, Gentili, Williams, Beck, Stumm (2016). Continuous deployment at Facebook and OANDA · Studie
- Śliwerski, Zimmermann, Zeller (2005). When do changes induce fixes? · Studie
- Vasilescu, Yu, Wang, Devanbu, Filkov (2015). Quality and productivity outcomes relating to continuous integration in GitHub · Studie
- Widder, Hilton, Kästner, Vasilescu (2019). A conceptual replication of continuous integration pain points in the context of Travis CI · Replikation
- Xu, Zhou (2015). Systems Approaches to Tackling Configuration Errors · Metaanalyse
- Yin, Ma, Zheng, Zhou, Bairavasundaram, Pasupathy (2011). An empirical study on configuration errors in commercial and open source systems · Studie
- Zhao, Serebrenik, Zhou, Filkov, Vasilescu (2017). The impact of continuous integration on other software development practices: A large-scale empirical study · Studie