Let's compare 3 different approaches to achieving better design:
| A/B Testing | Usability | Radical Innovation | |
|---|---|---|---|
| Cost | Low | Low–medium | High (unless lucky) |
| Benefits | 1–10% | 10–100% | 100–1,000% |
| Risk | None | Low | High |
| Who can do? | Everybody | Everybody | Geniuses |
| How often? | Weekly | Monthly | Every 10 years |
| GDP impact | Medium | High | Medium |
Definitions:
- A/B testing splits live traffic into two (or more) parts: most users see the standard design ("A"), but a small percentage sees an alternative design ("B"). After collecting statistically significant numbers, the design with the best KPI (key performance indicator, such as conversion or bounce rates) becomes the new standard. (One can test more than 2 design variations at a time through a related method called multivariant testing: for the sake of this article I'll lump these analytics methods together and talk about AB, regardless of the number of variables being measured.) As a form of web analytics — or intranet analytics — A/B testing has the huge advantage of testing live traffic to your website (or intranet).
- Usability refers to the full range of user-centered design (UCD) activities: user testing, field studies, parallel and iterative design, low-fidelity prototyping, competitive studies, and many other research methods. These types of studies usually require you to recruit test participants who should be representative of your target audience.
- Radical Design Innovation creates a completely new design that deviates from past designs rather than emerging from the hill-climbing methods used in more standard redesign projects. Example innovations include fundamental breakthroughs (the steam engine), new product categories (the locomotive), and major reconceptualizations of existing categories (the iPhone).
Cost
Incrementally, A/B testing is very cheap. You do need to pay a designer to create the "B" design, but most of the cost lies in the software to run and analyze the test, which is a one-time expense. Thus, if you're going to do it at all, you should run lots of A/B tests.
With usability methods, the costs range from $200 for a few quick activities to $38,000 to have a website analyzed by an independent expert. However, even the high end of usability costs pales in relation to the full cost of any enterprise-scale project. What's the budget to run a big-company website for a year? Easily a million when including staffing costs and overhead.
In contrast, radical innovation quickly runs into tens or hundreds of millions of dollars for fancy research labs and experimentation — and even then the vast majority of inventions go nowhere. Sure, a blinding insight occasionally creates a wonderful invention without the need for elaborate research. The invention of vulcanized rubber and penicillin are canonical examples of luck causing radical innovations. Still, luck favors the prepared mind, and Sir Alexander Fleming had already spent considerable time on fruitless bacteriological experiments when he stumbled across penicillin.
Benefits: From 1% to 1,000% Improvement
There's easily an order of magnitude difference in the effect size expected from each of the 3 design approaches:
- A/B testing usually identifies small improvements that might increase sales or other KPIs by a few percent. Sometimes you're lucky and get 10% or more. Website analytics' advantage is that it's the only way to reliably determine the best design approach when there's little difference between alternatives. Perhaps 1% doesn't sound like much, but if you could realize such gains every week for a year, you'd cash in more than 50%.
- In contrast, the full usability process typically doubles the benefit of the metrics you target (i.e., you get 100% improvement). For example, an enterprise software project might cut training costs in half or double employee productivity. If the old design was particularly poor, we sometimes see gains of 1,000% or more, but that's rare. More narrow usability efforts, such as fixing a particular design element, might result in a 10% gain for the desired usage metric.
- The sky is the limit for radical design innovation. Something sufficiently good could be 1,000% better than what went before. When defining a completely new product category, you can claim infinite gains, going from zero sales to some sales. However, more realistically, the new thing should be measured against the opportunity cost of business as usual, which surely is better than zero in any company that can afford to invest in advanced development.
Risk
There's virtually no risk in A/B testing: assuming that the statistical analysis is done correctly, there's close to 100% probability that you'll choose the design variation that makes the most money. If the difference between the two designs is small, you might have to wait a long time to collect enough traffic for statistical significance. But when the alternative design is only marginally better (if it's better at all), you lose little by sticking with the old design during a drawn-out test.
Usability methods also carry very low risk. Indeed, subjecting all designs to usability studies before shipping is prudent risk-management. Any terrible ideas that emerge from your fevered imagination will be shot down when confronted with real customers during user testing.
Radical innovation is extremely risky. Yes, you might invent the next iPhone. But you're more likely to invent the next Newton (Apple's early and doomed attempt at a personal digital assistant). In fact, almost all innovations fail. Even if it's a good idea, an innovation might be too early for the market. Pets.com, for example, is known mainly for its sock-puppet commercials during the dot-com bubble and for its spectacular bankruptcy, but other companies that eschewed the purported first-mover "advantage" are now making money selling pet food on the Internet.
Who Can Do It?
A/B testing can be done by a monkey (if the monkey has a graduate degree in statistics or can use stats software correctly). You don't need to understand design principles or user behavior. Simply try something different than your current design. If it scores better, keep it on the site. Otherwise, try something else.
Advanced usability methods often require trained specialists, but simple usability activities can be done by any member of a design team.
Returning to the theme of luck, it's certainly possible for below-genius personnel to be lucky and stumble upon a radical innovation. But typically, to systematically target radical innovation, you need to employ the best people in the world. Ask yourself whether you can realistically recruit, say, the top 1% of experts for your project. Even if you could, that's likely true only for 1% of projects. The remaining 99% of projects must make do with less exalted staff members, who can still be plenty talented and capable of more everyday advances — like the ones actually needed for most projects to succeed.
How Often Can Improvements Happen?
Websites with enough traffic can finish most A/B studies in a day or two. The main limiting factor is your ability to dream up new design variations. Also, designers need some time to realize each idea as an integrated user experience to give it a fair chance against the current site. But basically, there's no reason you shouldn't run a new A/B test every week.
Usability certainly can be done weekly as well. Since 1989, I've been evangelizing much faster and cheaper usability methods than most companies employ. (By comparison, Agile UX design is an upstart.) However, despite my best attempts at increasing the pace of usability, monthly turnaround is more common, so that's what I put in the table at the beginning of this article.
Realistically, for something to be truly radical, it can't happen too often. You can find a few exceptions that prove the rule, but most companies are happy to realize a radical innovation once every decade. (Most don't achieve this, even when they try; either their innovations end up being more modest or they fail completely.)
Economic Impact
Given the hugely bigger profit potential from a winning radical innovation, you might expect this approach to have the highest impact on the overall world economy. But all experience shows that most value is realized not from original breakthroughs but rather from the thousands of tweaks and implementations that build on that original work over subsequent decades.
Radical innovations soon become commodities: everybody has electricity, locomotives, and computers. What matters is what companies do with them: the railroad that invents a better way to ship grain might make more money than the company that invented the locomotive. Similarly, while the iPhone was a great advance, Android and other competitors are gradually eating market share.
On the other hand, because A/B testing usually results in tiny improvements, you might expect its overall contribution to be modest. Not so, since it's a method you can systematically apply again and again. Many small streams become a mighty river.
You can determine an approach's value by multiplying 3 factors:
V = G × F × N,
where
- V = total value, in terms of contribution to the economy's GDP.
- G = the gain from each improvement. Radical innovation totally wins this one.
- F = the frequency with which improvements can happen. A/B testing wins here, with usability as a close runner-up. Radical innovation is left several laps behind on this racetrack.
- N = the number of firms that participate in creating innovations. Again, the everyday methods win here because they can be employed by all companies, all the time. Working on payroll processing software? Usability will make it much better, but we'll probably wait 50 years for radical improvements in the product line. (Cloud-based processing might be somewhat better, but surely doesn't count as "radical" these days.)
Radical innovation is worth a lot when it happens, but it happens rarely and thus has only a medium-sized impact on the economy. A/B testing also probably has only a medium impact on the overall economy because many of its gains consist of moving market share among competing companies. Usability offers steady product improvements across a broad range, and its productivity gains are cumulative, resulting in a high potential for raising GDP.
The rough estimate in Tom Landauer's The Trouble with Computers is that GDP growth would increase by one percentage point if all companies employed sound usability methods. This may not sound like much, but it would add up to $2 trillion more in the United States alone over the next 10 years. (€1.5 trillion in the E.U.)
What to Choose?
Because usability makes the most money on average, it's the strategy that I recommend. (No surprise, if you've read my past articles.)
But, in truth, there's no reason to limit yourself to a single strategy because the 3 approaches complement each other well.
If you have the budget (or the luck), do try for radical innovation. But employ usability engineering to check whether your "innovation" is in fact any good before investing a fortune in bringing a failed product to market. And, once you have a product, refine it through both usability and A/B tests to ensure continuous quality improvement and stay ahead of the competition. In the long run, the cumulative effect of many small quality improvements is worth more than rare big breakthroughs.
