All too often, ideas that are truly innovative go against our experience and assumptions, or the conventional wisdom says Stefan Thomke. It’s time to be more experimental in the business world
Innovation is important because it drives profitable growth and creates shareholder value. But here is the dilemma: despite being awash in information coming from every direction, today’s managers operate in an uncertain world where they lack the right data to inform strategic and tactical decisions.
Consequently, for better or worse, our actions tend to rely on experience, intuition, and beliefs. But this all too often doesn’t work. And all too often, we discover that ideas that are truly innovative go against our experience and assumptions, or the conventional wisdom. Whether it’s improving customer experiences, trying out new business models, or developing new products and services, even the most experienced managers are often wrong, whether they like it or not.
Business experiments raise the innovation game dramatically
In 2012, a Microsoft employee working on Bing had an idea about changing the way the search engine displayed ad headlines. The change wouldn’t require much effort, but it was one of hundreds of ideas proposed, and the program managers deemed it a low priority. So it languished for more than six months, until an engineer launched a simple online controlled experiment to assess its impact. Within hours, the new headline variation was producing abnormally high revenue, triggering a ‘too good to be true’ alert. An analysis showed that the change had increased revenue by an astonishing 12% – which on an annual basis would come to more than $100 million in the United States alone – without hurting key user-experience metrics. It was the best revenue-generating idea in Bing’s history.
This story illustrates how difficult it can be to assess the potential of new ideas. At first blush, the original idea was dismissed as unimportant by managers. What made the difference? The ability of an employee to launch a rigorous experiment to assess the idea’s impact.
Companies that annually run more than ten thousand online experiments, which individually engage millions of users, have discovered that an “everything is a test” mentality yields surprisingly large payoffs and competitive benefits, and may even help stock performance.
These ‘experimentation organisations’ are companies in which experimentation is embraced by every employee, from top to bottom. Experimentation is not simply the responsibility of a department, an R&D laboratory, or a group of specialists. Rather, all employees participate in one way or another in an organisation in which doing experiments is as salient as running the numbers. The organisation’s ethos is to think experimentally.
For example, at Booking.com, the world’s leading accommodations platform, all employees can define a hypothesis and launch an experiment on millions of users without permission from management. The company spent more than a decade building a democratic culture in which B2B and B2C experiments are deeply engrained in its daily routines. On any day, its staff runs more than one thousand rigorous, concurrent tests on its website, servers, and apps to optimise experiences. With quadrillions (millions of billions) of landing-page variations, customers booking a room on its website are all part of the company’s experimentation ecosystem.
Continual experimentation
Orienting a business toward the practice of continual experimentation does not guarantee that each experiment will be a success or that all will be perfect on the first go-round. Experimentation itself does not automatically assure riches. Indeed, the failure rate of experiments can be 90 percent or higher—whether they are conducted by a single scientist, a world-famous laboratory, a marketing department, or business strategists.
But if the single experiment does not work in itself, it generates information: Why it didn’t work; what were we wrongly assuming; where the design, implementation, etc went awry. And, most importantly: What can we learn for the next experiment? Of course, learning from success and failure has always been intrinsic to experimenting. What has changed is that we now have the tools that allow us to learn about innovation performance—new products, customer experiences, and business models—rapidly, inexpensively, and at an unprecedented scale. Large-scale experimentation means that companies can systematically kiss a lot of frogs to (hopefully) find a prince.
Many managers mistakenly believe that experimentation is about throwing many ideas against the proverbial wall to see what sticks—a technique known as ‘spray and pray’. That’s not at all what happens in disciplined testing, which can isolate variables and establish cause and effect. If anything, experimentation organisations have a laser-like focus on the things that truly matter and don’t rely as much on experience, past data, intuition, imitation, or so-called best practices.
Consider the following example, from Booking.com which was pivotal in the company’s coming to grips with the perils of business assumptions, and the realisation of the power of experimentation.
Early on in its history, Booking discovered that it could not rely on intuition and assumptions, as a manager told me: ‘We see evidence every day that people are terrible at guessing. Our predictions of how customers behave are wrong nine out of ten times.’ For instance, it was assumed that customers would want hotel offers packaged with other products. Why? Because travel brochures – the kinds we used to pick up at travel agencies – did such packaging. Booking is an online service, however.
Does what works in a travel agency work on a website? Likewise, it was just assumed that customers would want a chat line they could go to while they were working through the online booking process. Why? Because it was just assumed they would—other companies have chat lines, after all. It was also assumed that customers would visit videos of potential travel destinations as they browsed the website. Why? Well, it just seemed intuitively obvious.
Experimentation empowered
None of this turned out to be the case. How do we know? The discovery that these assumptions, guesses, or intuitions were not applicable to Booking’s customers was the work of ‘experimentation empowered’ employees of the company, all of whom, through a documented process could design experiments challenging those assumptions— and many more.
The Booking example is an exciting and illuminating insight into business experimentation and why it is of absolute importance to a company’s ability to compete. Experimentation helps us begin to answer the kinds of questions that all organisations confront, from what products to make and what customer experiences to offer, to what information we need to make those decisions.
Stefan Thomke is the William Barclay Harding Professor of Business Administration at Harvard Business School.
Reprinted by permission of Harvard Business Review Press. Excerpted from Experimentation Works: The Surprising Power of Business Experiments. Copyright 2020 Stefan H. Thomke. All rights reserved.
