Rebuttal to Critics of Kindle Fire Usability Study

by Jakob Nielsen on December 5, 2011

Sidebar to Jakob Nielsen's column on Kindle Fire Usability Findings, December 2011  

Our recent usability study of Amazon.com's new Kindle Fire has generated thousands of reactions on the Internet: Tweets galore, blog postings, and reader comments on the press coverage at various news sites. It's the beauty of the Internet that it's an equal-opportunity publishing environment that allows everybody a voice. Many of the comments were of the nature "I own a Kindle Fire, and my experience matches Nielsen Norman Group's research." Others were honest differences of opinion based on different use cases, such as "I like having a cheap tablet to watch movies, and I don't browse the Web much, so any problems in the browser don't bother me."

There were also the usual enemies of usability whose profanity-laced jeering of anything posted by me or other serious user researchers is rather old hat. I ignore these people. If empirical facts don't convince them, then nothing I can add will do the trick. Let them eat cake and spend their time in mutual admiration of pretty but useless sites.

However some comments were more substantial disagreements with our research and deserve a rebuttal.

"This Is Only Version 1.0; Later Versions Will be Better"

No doubt true. The New York Times reports that Amazon is already planning a software upgrade to be released soon. (Of course, promises of better things to come is a strong argument for holding off on any purchase until these updates have actually been made and we know whether they fixed the usability problems or not.)

The day after I posted my original column, I got an email from Google saying that they had already changed their page design, when served to Kindle Fire users, to overcome many of the usability problems I pointed out. To which I can only say, "good work, Google guys."

Thus, if we reran our study now, we would already expect the outcome to be slightly better, to the extent that it's now easier to use an external search engine. (Assuming that users use Google, but most do. We don't point them to any specific search engine but sit back and watch where people go on their own initiative.)

Google is the world's wealthiest website and has many good UX people and strong developers on board who can make appropriate UI changes fast. Most websites will be slower. Also, as pointed out in my analysis, the economic case will not be there for a broader range of sites to adapt their design to 7-inch tablets unless this platform becomes incredibly popular.

The total user experience for a platform is comprised of both its native UI and of the design of content and services projected onto the device. Thus, if most sites are slow to change, the Kindle Fire web experience will also only improve slowly.

Kindle Fire feels like a product that was rushed into production. A few extra months of usability studies, iterative design, and more sophisticated programming would have worked wonders that are hopefully going to ship later. I understand why Amazon might want to ship a poor product in late November rather than a good product in February: they want to catch the holiday shopping season. Whether the extra sales are worth the brand damage from a low-quality user experience is difficult to judge.

When writing articles, I luckily don't have to face the tradeoff between earlier sales and a better product. I simply reveal what the usability studies tell us. My job is to tell the truth, not to manage product development. (A consulting project is very different from writing a website column: when advising clients, one does have to get into the inevitable tradeoffs of any development project.)

One final reason to report on the usability weaknesses of the Kindle Fire as it was originally released: executives in other companies often instruct their design teams to copy the work of big and famous companies under the assumption that the big boys must be right. In a case like Kindle Fire, the biggest boy (Amazon.com) committed a lot of design sins that would hurt other companies if they were to emulate these design decisions. Amazon may have had its reasons; it may be able to make its money back; it may have improvements planned. None of these points make the original Fire design worth copying for companies in less privileged circumstances.

"What do You Expect for $200?"

I fully realize that not everybody is a well-paid member of the Silicon Valley elite with $500 in spare change to drop at any new gadget to come along. Cheap products are definitely needed in order to reach a broad audience. I simply don't think that poor people should get shoddy products. Of course, they can't expect luxury at a cutthroat price, but they should get a certain base level of quality at any price.

As an analogy, if you want a cheap hotel room in Tokyo, it'll be tiny, the location won't be right in the Ginza, the staff will have very limited English proficiency, and there may be nothing but raw fish served for breakfast. Even so, the room will be spotlessly clean and the staff will be extraordinarily friendly and help you as much as they can to the limit of their language skills.

Similarly, a cheaper 7-inch tablet obviously can't show as big beautiful photos or detailed videos as a 10-inch tablet. There will be less info on each screenfull, necessitating more navigation. Some higher-end apps may not run. There's no camera. All this is fine and the penalty you pay to get a cheaper device.

However, there is no reason that the user interface design should be bad and not optimized for the available screen space. Creating a vastly better user experience would have been a simple matter of having a few more usability folks, interaction designers, and programmers devoted to optimizing the user interface. Total budget: less than a million dollars, or about 10 cents for each unit they expect to sell during the first year.

The difference between user interface design and hardware specs is that better usability is derived from one-time expenses for user studies, design iterations, and coding — whereas beefier hardware (say, adding a camera) is a repeated expense for each additional unit manufactured.

This means that even cheap devices can have great usability because the cost of better research and design is amortized across millions of devices. This is why usability has stupendously high ROI for any big project.

In the hotel analogy, if you want twice as large hotel rooms, then the hotel's real estate costs will approximately double, as will costs for cleaning, heating, etc. You can't offer big rooms cheaply. But friendly front-desk staff might not cost much more than sourly people. And for sure, if you have a hotel chain , the user experience of online registration can be as smooth as you want, because some extra rounds of iterative design amount to a few cents for each of millions of room nights. (Usually, guests wouldn't even pay these few cents, because the higher price of a better website would be recovered many times over by increased sales.)

"How Can You Complain That Fire Is Heavy When It Weighs 2/3 of the iPad"

Yes, the Kindle Fire weighs less than an iPad. But as we found in the first iPad usability studies, users also find the iPad too heavy.

More to the point, iPad use is often two-handed, whereas the hope was that smaller 7-inch tablets could be used single-handedly. (For sure, the old Kindle was light enough that one could read all evening while holding it in one hand.) Distributing 2/3 of the weight across 1/2 of the muscles results in higher strain on those muscles that are recruited to hold the device. Just like lifting a 40 pound dumbbell with one arm is harder than lifting a 60 pound barbell with both arms.

"You Are Confusing an E-Reader with a Tablet"

If the Fire continues to offer terrible web browsing, then it may indeed end up as nothing but an e-reader. But the hope was surely that 7-inch tablets would be a new class of broader-use device, populating the ecological niche between 10-inch tablets and mobile phones.

Also, even as an e-reader the current user experience fails, because of the difficulty in navigating magazines and other non-linear content.

"Amazon Will Sell Millions of Kindle Fire Despite its Usability Deficiencies"

History suggests that usability is only one of many factors to influence product success. For example, PCs with terrible MS-DOS outsold Macintosh by huge margins in the 1980s, and primitive Windows user interfaces continued to outsell better Mac designs in the 1990s. By now, Windows 7 is pretty good, but there was a long time when PCs sold because of low price and strong application availability, not because of the OS UX.

Similarly, Kindle Fire could sell well despite its design weaknesses. Marketing reach counts for a lot, as does the cheaper price. Also, the one place where Fire has great usability is for shopping at Amazon: both when using the built-in app to shop for physical products and when using the integrated features to download (and easily pay for) content. Thus, Amazon might end up scoring a good profit from the device, even if it's not a big success.

Still, even though it's easy to buy and download content, users will ultimately be dissatisfied when it turns out that it's unpleasant to use the content (at least for magazines, newspapers, and textbooks).

"Jakob Nielsen Hates All User Interfaces Anyway"

Not true. There are many designs I find to be quite nice and use every day. If anything, my failing is that I often get overly excited about the promise of new technology. That's why I was very hopeful about the Kindle Fire before we got around to testing it. (In fact, we pre-ordered 6 copies for Nielsen Norman Group.)

Also, I am a big fan of the pre-Fire Kindles which scored very well in our study of people reading from tablets. Two years ago, I gave every member of my team a Kindle as a holiday present.

Admittedly, my published writing does cover more bad design than good design. This is because a website has to focus on polemic and hard-hitting content to get visibility. Bland content doesn't get read. That's a big difference from other media forms, such as our training seminars, where we also include prescriptive examples of good design. When you have people trapped in a lecture room all day, you have the luxury of a much broader set of communications strategies than when trying to attract Internet clicks and avoid losing impatient web readers.

Many times, I have praised good design — for example in the Intranet Design Annual or the Application Design Showcase.

I was thrilled with the Kinect last year, even though readers of my analysis might have fixated more on my reports of its usability problems than on my praise of its good usability aspects. The facts are that most user interfaces have some good and some bad aspects, but whatever I write about the problems gets more attention. Thus, many commentators summarized my Kinect analysis as "Nielsen condemns Kinect usability," whereas I actually praised its promise while pointing out some usability problems with that year's crop of gestural games.

"You're a Typical Apple Fanboy"

I'm not. I admit to having been a loyal Mac user for 12 years (1986–1998), but in 1998 I gave up on the Macintosh. I was starting a new company and didn't feel safe basing it on a platform that was fast going down the drain. A few years later Apple was rescued, but by then Nielsen Norman Group had endless documents and software dedicated to Windows.

I certainly don't hate Amazon.com. I have been a loyal customer since 1996.

"Your Site Is Ugly"

Yes, and so what? Usability and visual design are different (though related) things.

I explained the reasoning behind the useit.com design in an interview with The Guardian newspaper. Briefly, it's aiming to hit people on the reflective level of Don Norman's theory for emotional design.

"The Study Sample Size Was Small"

I am gratified that very few commentators made this specious argument. Maybe the deeper insight one gets from qualitative research is starting to get more respect.

There are many, many things one cannot reliably conclude from a small sample size. For example, even if we observed 3 of 4 users doing X, it would be unfounded to report that "75% of users do X." Similarly, we couldn't report that "mobile sites were Y% easier to use than full desktop sites on the Kindle Fire." The confidence interval around any estimated Y-value would be too huge for this statement to be meaningful.

But it's perfectly valid to report trends and conceptual insights. For example, mobile sites did do better, even if we don't know exactly how much better. I can be confident in reporting this finding because I can explain the empirical observations with existing knowledge from a vastly larger base of test results with mobile devices and tablets.

We've tested hundreds of mobile/touch designs (for several years now) with users on 4 continents and a broad range of devices. (In fact, our first mobile usability research was in 2000.) When insights from this large amount of research match conclusions from a small study, it's reasonable to believe the small study. For example, our findings regarding the fat-finger problem on Kindle Fire are almost certainly true, even though they are based on a few users with a small range of hand sizes.

On the other hand, let's hypothetically say that we had observed a "skinny finger" problem with the Kindle Fire that was in complete opposition to everything we had previously learned about touchscreen usability. I would not have felt confident in reporting such a revolutionary finding without further research going beyond the initial small study.

"The B&N Nook is Much Better: Why Didn't You Test That?"

Some people do seem to prefer the Nook. We picked the Kindle Fire because it was the most ambitious project for designing a high-powered 7-inch tablet. For example, the new Silk browser has an interesting proxy-based architecture that was worth testing.

We couldn't test two tablets in this project due to the cost of running empirical usability studies. The purchase price of a Nook is nothing. But the cost of planning, running, and analyzing actual sessions with real test users is substantial.

The big difference between our 7-inch project and those reviews you read on technology blogs and the like is that we collect actual empirical evidence of observed user behavior, as opposed to relying solely on our personal opinions. This approach leads to better insights into usability issues and user experience strategy, but is much more resource-intensive, meaning that we end up testing fewer devices. (This, by the way, is also a key difference between our usability conference and most other design events: we talk about what actually works — or doesn't work — in testing with real users, not about what we personally like or think is cool.)

Press Coverage

Some of the press reactions to the Kindle Fire usability study:


Share this article: Twitter | LinkedIn | Google+ | Email