Software Engineering, geprüft · Teil 2 von 13
Decide: most ideas fail
At Microsoft, a new feature doesn't simply ship. It goes into an experiment. A slice of users gets the new version, the rest keep the old one, and the platform measures whether the change moved the number it was built to move, such as clicks or revenue. Over the years Ronny Kohavi and his colleagues ran thousands of these controlled experiments, and in 2013 they reported what they had learned.
Only one third of the ideas tested at Microsoft improved the metric they were designed to improve. At Bing, a product already tuned hard, success was even rarer ~85 %. These were not random ideas. Each one had been proposed, argued for, prioritised and built by people who believed in it. Their conclusion was blunt: "we are poor at assessing the value of ideas."
That is an uncomfortable starting point for a part about deciding what to build. Most of software engineering's folklore about requirements and process assumes that the hard part is building the thing right. The evidence keeps suggesting that the harder part is knowing what the right thing is. The question is plain. Do we know what to build, and do we know how to work while we build it?
Most ideas don't work
Kohavi's team put it carefully: features are built because teams believe they are useful, yet in many domains most ideas fail to improve key metrics. Two out of three is not a story about bad teams. It is the base rate.
The usual defence against a bad idea is an experienced person saying no. Kohavi has a name for that: the HiPPO, the Highest Paid Person's Opinion. His group's experience was that learning and return on investment came when teams listened to their customers rather than the HiPPO, and the experiments repeatedly stopped features that key stakeholders had been excited about ~15 %. Seniority doesn't fix a judgement nobody can make reliably.
The opposite folklore is just as popular: the button colour that lifted revenue by millions. Such wins happen, but at Bing they were extremely rare, and the experimenters' rule of thumb was that most amazing results turn out to be false when reviewed carefully ~15 %. The honest picture is duller than either story. Most changes do nothing, a few help, some hurt, and you can't tell which in a meeting.
This is the third thread of the series at its cleanest. What engineers and managers believe about their ideas, and what gets measured when the ideas meet users, disagree two times in three.
Blaming the requirements
When projects fail, practitioners blame the requirements. One survey, the NaPiRE study led by Daniel Méndez Fernández, asked practitioners in 228 companies what goes wrong. Incomplete or hidden requirements was the only problem consistently in the top three, followed by communication flaws between the team and the customer ~70 %. The diagnosis is plausible, and it is a diagnosis by the people involved. It measures perception. The stronger line, that bad requirements are the single biggest cause of failure, leans on consultancy data.
A related belief says most defects are born in the requirements. In safety-critical embedded systems, Robyn Lutz found that safety-related errors arose most commonly from gaps between the documented requirements and what the system actually needed. But a study of large fault and failure datasets by Maggie Hamill and Katerina Goseva-Popstojanova found requirements faults to be one of three most common types, alongside coding faults and data problems, and noted that "contrary to the popular belief" a significant share of failures came from late life-cycle activities ~50 %. Requirements matter a great deal. They don't carry all the blame.
Read side by side with Kohavi, the requirements literature looks slightly different. Incomplete requirements and poor communication with the customer are what you'd expect if nobody can tell in advance which ideas will work. The problem isn't only that requirements are written badly. It's that they are guesses.
Ask the users, show them something
If the team can't judge an idea, the obvious move is to bring in the people it's for. Muneera Bano and Didar Zowghi reviewed 87 studies of user involvement. 52 found it contributed positively to system success, 12 found a negative contribution and 23 were uncertain. Their summary: it is a double-edged sword, and if not managed carefully it may cause more problems than benefits ~70 %. Extreme Programming took involvement furthest with a customer on site full time. Studies of it found the practice achieving excellent results and burning out the customers doing it ~30 %. Software is people talking, and that includes the people paying for it.
The other move is to stop describing and start showing. In 1984, Barry Boehm, Terence Gray and Thomas Seewaldt had seven teams build the same small application, some writing a specification first and some prototyping. The prototyped products performed about as well with roughly 40% less code and 45% less effort. They rated lower on functionality and robustness, and higher on ease of use and learning ~30 %. Seven teams and one small product is a thin base for the claim it usually supports, and nobody has repeated it at scale. The trade-off it shows is more interesting than the headline anyway: a prototype gets the user's experience right sooner and the edge cases later.
Where did the waterfall come from?
Every agile talk needs a villain, and the villain has a birth certificate. In 1970, Winston Royce published a paper called "Managing the development of large software systems". Early in it sits the diagram everyone knows: system requirements, software requirements, analysis, program design, coding, testing and operations, each box stepping down to the next like water over rocks. Generations of textbooks reproduced it as the waterfall model, the way software was supposed to be built, and later as the way it was supposed to stop being built.
Now read the rest of the paper. Royce presented that single pass as the simplest description, and said that it would not work for all but the most straightforward projects. The diagrams that follow grow arrows pointing backwards. Then comes his actual recommendation, which Craig Larman and Victor Basili quote in their history of iterative development: do it twice. Build a first version, learn from it, then build the real one ~15 %. As Larman and Basili put it, many incorrectly view Royce's paper as the paragon of single-pass waterfall. He never even called it a waterfall; the name came later.
The diagram travelled; the pages after it didn't. It is the same pattern as the 100x curve in part 1, with a picture in place of a number. Five years later Fred Brooks gave Royce's advice its famous form in The Mythical Man-Month: plan to throw one away.
The second half of the myth is that iteration was invented by the agile movement. Larman and Basili trace iterative and incremental development on real projects back to the mid-1950s, and found prominent voices supporting it in every decade since, with many large projects using it successfully ~5 %. The Agile Manifesto of 2001 gave the practice a name, a movement and a fight. It didn't give it a beginning. The fight was against a method that its supposed author had warned against thirty years earlier.
What the agile evidence actually says
None of this means agile was wrong. It means the case for it was made by argument, and the evidence arrived later and thinner. In 2008, Tore Dybå and Torgeir Dingsøyr published the main systematic review of empirical studies of agile development. Their conclusion was that the strength of the evidence for the benefits and limitations of agile methods was very low ~30 %. By then the method had already spread far faster than the evidence.
On productivity, the studies they found point both ways. One reported a 42% increase for an agile team, another a 44% decrease for an XP team against a traditional one, and an apparent 337% gain turned out to be an XP team writing 3.5 times as much code without delivering more functionality ~50 %. That last one is a lesson about measuring output in lines of code as much as about agile.
The more recent evidence is better news for agile, with a caveat. Magne Jørgensen surveyed software projects and found that agile projects succeeded more often, including large ones ~70 %. That cuts against the folklore that agile doesn't scale ~30 %. But the projects were self-reported and the analysis correlational, and Jørgensen's own candidates for the reason are specific: flexible scope, frequent deliveries to production, and more competent providers. Those are practices, not a label.
The same goes for the more doctrinaire readings. In a grounded-theory study, Michael Waterman, James Noble and George Allan found that agile teams decide how much architecture to design up front by weighing their context. Teams that spent too little time on it faced more risk and a higher chance of failure ~30 %. Architecture doesn't simply emerge. Royce would have recognised the argument.
The rituals, measured
Scrum's daily stand-up is close to universal, and it is one of the few rituals anyone has studied closely. Viktoria Stray and colleagues surveyed developers and found them neutral on average, which hides a split: most respondents were either positive or negative. Junior developers liked it; senior developers and members of large teams liked it least. Badly run stand-ups reduced job satisfaction, trust between colleagues and well-being ~50 %.
Then they watched 15 teams. On average, only 34% of the meeting went on the three Scrum questions. The rest was largely problem-solving, the thing the guidance says to take offline, and in interviews it was the part people most often named as valuable ~30 %. The canonical first-thing-in-the-morning slot fared no better: an early start often reduced the meeting's value, and many preferred meeting just before lunch, where it interrupts nothing and the team can go and eat together ~30 %. The stand-ups that worked broke the rules. They worked because people talked to each other about the problems they had.
Retrospectives are meant to be how a team improves itself. Timo Lehtinen and colleagues studied 37 retrospectives over three years. Some led to effective improvements. Others kept circling the same topics, such as the number of known bugs, and discussions not backed by evidence tended to reflect the strongest opinions in the room ~50 %. A retrospective is an experiment without a measurement, which brings this part back to where it started.
What doesn't work
- Trusting the HiPPO. Experienced people misjudge which features will pay off as often as anyone; the experiments keep stopping their favourites ~15 %.
- Hunting for the magic button. Big wins from tiny changes are extremely rare, and most amazing results don't survive review ~15 %.
- Blaming Royce for the waterfall. He drew the single pass to say it rarely works, and told readers to build it twice ~15 %.
- Calling agile proven. The main review graded the evidence very low; later surveys are correlational ~30 %.
- Running the stand-up by the book. Three questions, first thing, no problem-solving: the evidence is against all three ~30 %.
Try this
- Assume your idea will fail. Decide before building what metric it should move and how you'll know, then check. Two times in three, it won't.
- Put a version in front of users early. A prototype or a small release answers questions that a requirements meeting can only argue about.
- Plan to build it twice. Budget for a first version you'll learn from. Royce said so in 1970, and Brooks in 1975.
- Size the up-front design to the risk. Neither a big design phase nor none at all; more where mistakes are expensive to undo.
- Let the stand-up solve problems. Pick a time that interrupts no one, keep it short, and stop if the team, not the guide, says it isn't worth it.
19 Behauptungen geprüft
Die Belege hinter diesem Artikel
Jede Behauptung oben, bewertet nach der Chance, dass sie der Forschung standhält. Öffne eine für die Quellen und ihren genauen Wortlaut.
- Bano, Zowghi (2015). A systematic review on the relationship between user involvement and system success · Metaanalyse
- Boehm, Gray, Seewaldt (1984). Prototyping Versus Specifying: A Multiproject Experiment · Studie
- Dybå, Dingsøyr (2008). Empirical studies of agile software development: A systematic review · Metaanalyse
- Fernández, Wagner, Kalinowski, Felderer, Mafra, Vetrò, Conte, Christiansson, Greer, Lassenius, Männistö, Nayabi, Oivo, Penzenstadler, Pfahl, Prikladnicki, Ruhe, Schekelmann, Sen, Spinola, Tuzcu, de la Vara, Wieringa (2016). Naming the pain in requirements engineering · Studie
- Hamill, Goseva-Popstojanova (2009). Common Trends in Software Fault and Failure Data · Studie
- Hofmann, Lehner (2001). Requirements engineering as a success factor in software projects · Studie
- Jorgensen (2019). Relationships Between Project Size, Agile Practices, and Successful Software Development: Results and Analysis · Studie
- Kohavi, Deng, Frasca, Walker, Xu, Pohlmann (2013). Online controlled experiments at large scale · Studie
- Kohavi, Longbotham, Sommerfield, Henne (2008). Controlled experiments on the web: survey and practical guide · Studie
- Larman, Basili (2003). Iterative and incremental developments. a brief history · Quelle
- Lehtinen, Itkonen, Lassenius (2016). Recurring opinions or productive improvements—what agile teams actually discuss in retrospectives · Studie
- Lutz (1993). Analyzing software requirements errors in safety-critical, embedded systems · Studie
- Stray, Moe, Bergersen (2017). Are Daily Stand-up Meetings Valuable? A Survey of Developers in Software Teams · Studie
- Stray, Moe, Sjoberg (2020). Daily Stand-Up Meetings: Start Breaking the Rules · Studie
- Waterman, Noble, Allan (2015). How Much Up-Front? A Grounded theory of Agile Architecture · Studie