Summary: Usability tests of 46 Flash applications identified basic issues related to the ephemeral nature of Web-embedded apps. Some findings restate old truths about GUIs; others reflect the Net's new status as nexus of the user experience.
The Internet is changing. Although people have primarily used it to read email and Web pages, more functionality-oriented applications are now emerging, with the goal of providing new features that do more for users. Developers are creating many of these applications using Macromedia Flash, because traditional Web pages are better suited to what they were invented for — reading articles — than to the new goal of manipulating data objects.
Although it's possible to provide Web-enabled applications through standard Web pages and the traditional browser interface, doing so neglects twenty years of GUI advances and harkens back to the asynchronous interaction style last seen in the '70s on IBM 3270 terminals. It can be done — and it has been done — but it's not good for usability.
The alternative? Embed the new functionality within a GUI, as with traditional shrink-wrapped software for the Apple Macintosh or Microsoft Windows. Flash offers designers a way to create GUIs for Internet applications, making interaction styles that support functionality beyond reading and browsing possible.
This possibility sounds good. However, in the usability field, we've learned that more technical capabilities and a broader set of design options usually translate into more rope for hanging the users. Designers almost always use new features to excess, and it takes some time to discover the most appropriate way of applying new technology to suit human needs.
Running user studies of early applications can accelerate this adaptation by helping us identify usability pitfalls and derive design guidelines to codify best practices. We have done exactly that. We tested 46 different Flash applications with users in the United States, Germany, and Japan, and summarized the resulting lessons for Flash applications in 117 usability guidelines.
Our biggest finding may be this: Most current Web-based applications are ephemeral and must be immediately understandable or users will fail. Usability requirements for Web applications are far stricter than they ever were for traditional software.
We calculated people's ability to use the Flash applications based on two different ways of treating partial success. In our feature-success calculation, we examined participants' ability to use Flash application features and gave partial credit when users completed some parts of a task, even if they failed on other parts (and thus did not accomplish the full task). If users required assistance to proceed through part of a task, we gave them no score for that part, but we gave them credit for anything they figured out on their own. With this feature-based scoring method, users had an average success rate of 64% across the 46 designs.
We also computed a task-success score, where no credit was given if users failed to complete the task, even if they used some features correctly. If, as often happened, users couldn't navigate from the homepage to the application, we gave them a task score of zero. We did the same if they failed on any other step. The theory behind this calculation is that either they did it or they didn't. With this stricter scoring method, users had an average success rate of 45% across the 46 designs.
Which success rate is most representative of Flash application usability? You can argue for either, which is why we computed both.
The task-success score is the best indicator of whether the total user experience, which integrates traditional HTML Web pages and Flash functionality, is working as intended. After all, most users do not have a kindly usability facilitator sitting next to them, so if they can't overcome difficulties, they'll typically leave the site. We often helped users overcome difficulties in getting into the application in the first place. Without such assistance, they would not have reached the application, and thus any success they might have had using its features in our study would not have occurred in normal usage.
The feature-success score (which gives credit for partial task success) might be the best at assessing Flash design quality, in isolation from the host website.
Web-Based Applications are Ephemeral Applications
In principle, Flash is just an implementation technology. It could be used to build traditional desktop applications, which would have the same usability problems as traditional computer software. However, all of the 46 Flash applications we tested were Web-based in one form or another.
In our study, the low success rates are directly related to the fact that the Flash applications were Web-based. Web applications differ from traditional applications in several ways, and users view them as low-commitment transient encounters — a concept we refer to as "ephemeral use."
- A Web-based application is usually a website component, meaning that users must navigate from a traditional, information-oriented Web page to the Flash application's functionality-oriented tool. In our study, users failed to make this initial step 36% of the time, which is one of the main reasons for the low task-success rate.
- Once users find the application, they must understand what it does and what it can accomplish for them, as well as its general task flow and structure. This is true for all software, but most traditional software includes initial training that establishes the basics. Also, many software products are well known and users know their basic purpose even before installing them. This is true for both enterprise systems (such as intranet-based expense reporting or time sheets) and desktop software (such as PowerPoint). In contrast, users are thrown directly into a Flash application from the website, often unexpectedly.
- Users have less motivation to understand advanced features in Web-based applications, because the applications are not typically a core part of their work. In contrast, many jobs demand traditional software use, either as a defining aspect of someone's work (as with airline reservation agents, for example) or to generate the deliverables that measure employee performance (as is often the case with Microsoft Office).
- Users rarely return to the same Flash application multiple times, so they'll rarely benefit from a build-up of learning about a specific GUI. In contrast, traditional software is often used repeatedly by the same person.
Because remote servers host Web-based applications, users might have to register and/or sign in to be able to manipulate their own data. Storing and retrieving user data is an added complication because, unlike traditional software data, Flash data can't be stored on the user's machine.
The fundamental issue behind most of these points is that Web-based applications are ephemeral experiences that users encounter in their surfing. They might use the application only once, which means that the immediate user experience is all that matters. Even when users access a Web-based application several times, their experience is typically intermittent and brief.
We should emphasize that the ephemeral nature of these applications is not a fundamental characteristic of Flash; instead it's a characteristic of the applications implemented in Flash when we ran our studies. However, the Web encourages a much broader range of applications than traditional software. This diversity and breadth of experience is positive, but because people spend less time with each application, there are usability implications.
Ephemeral applications — and thus most Flash applications — must be simple; they cannot have too many features. This streamlining requires that developers understand users' needs even more than usual, because they can't simply throw a huge feature set at users and have them puzzle out what they need to accomplish something useful.
Ephemeral applications must also give users a quick overview of their features and basic task flow. Such applications are primarily accessed by first-time users, who must build a conceptual model of how to deal with them. Yet, at the same time, users won't invest a lot of time exploring or learning the application, because they know they may never return.
Navigating to Web-Embedded Applications
Many applications that are embedded within websites are linked in ways that cause most users to avoid them. In our study, users in the U.S. had to be directed into the application area 36% of the time. Normally, then (lacking a test facilitator's help), more than a third of users would never launch such linked applications, even when they wanted to perform the tasks that the applications support.
Users' frequent inability to find applications significantly lowered task-based success scores in our study. Basically, companies can get 56% more return on their application investments if designers simply make the applications easier to locate. Of course, we would also like companies to improve the usability of the application itself, but lacking this, they can increase usage by more than half if they simply ensure that all users can find the application.
So why the problem? Designers are so happy with their applications that they over-promote them on Web pages. It's common for designers to link to an application through a big colorful box of graphics that looks suspiciously like an advertisement. As we've seen in tests for many years now, Web users have developed a strong tendency to ignore anything that looks like an ad. This banner blindness serves users well when navigating traditional websites, because it lets them focus on the useful links and ignore the ads. Unfortunately, if the colorful promotion links to something useful, like an application, users will never find it, because they simply won't read the box — much less click it.
Some links to applications use animated words in an attempt to appear even more attractive and promote the application's various benefits. This technique backfires even more, because users firmly believe that anything containing moving or blinking words is bound to be a useless advertisement. This belief is typically true, and saves users much time once they've developed the ability to ignore moving text.
Two examples from our study:
- Pergo sells laminated floor covering. On their site, we asked users to find the supplies they'd need to install a new kitchen floor. One user was a particularly sad case: On a page that asked for the square footage of the area to be covered, he was swearing as he tried to calculate his floor area by hand. Next to the form he was struggling with was a large animated graphic with flying words, including "room planner," "set up room size," "length," "width," and several other terms indicating that the box linked to an application for computing floor sizes. Too bad this user didn't see it. Nor did our other test users. To get usability data about the actual interaction design, we had to force people to launch the application.
- The Haribon Foundation in the Philippines had lots of good information about the environment and endangered animals. On the foundation website, we asked users to find out which endangered birds live in a certain location. Despite the fact that every single page on Haribon featured a big colorful box promoting an interactive map of key conservation sites in the Philippines, nobody clicked this link. People tried every other alternative to find out about the birds, and again we had to force users to click the application link before we could collect data for our application design guidelines.
The problem is clear: users try to avoid anything that's overly hyped or promoted, especially if it looks like an advertisement. They also don't care about interactive tools for their own sake; users care only about their own problems and finding a straight task flow that will solve them. Users will ignore anything that sounds complicated or computer-like.
The solution is also clear: Even if you have an advanced application, don't tell users. Simply link to the application from your website, using a standard hypertext link. The more ordinary the application sounds and the more you present it as a solution to users' problems (rather than showing off your technological prowess), the more clicks you will get. Name the link something that clearly indicates what the application does. Avoid hype, and don't tell users that it's interactive or built in Flash.
Object-Oriented GUI Design
Conducting user sessions with Flash applications was in many ways reminiscent of the 1980s when we tested the first crop of Macintosh applications. Many of the usability problems identified in our Flash study are related to basic GUI concepts, such as making controls obvious and easy to grab. One of our Flash guidelines is a virtual copy of a finding from the 1980s: You must provide generous click zones around active screen areas, or users will think that they clicked something even when the computer's strict definition of clickable pixels says they didn't.
We also repeated a finding from early presentation package studies: When you create new objects on a drawing canvas, they should be staggered relative to other objects so that they're all visible. Other Flash guidelines are new and would never be found in relation to traditional software. For example, we discovered many usability issues relating to sound and animated objects, both positive (you can use them to indicate change and direction) and negative (they can be distracting, annoying, and harmful for users with disabilities). Some of the early Flash applications have apparently inherited bad habits from Web design.
One particular usability problem is worth emphasizing: in several applications, users missed many of the options because of nonstandard scrollbars. A scrolling control is a standard user interface element in application design, and it should be designed in accordance with users' expectations. We did see a few nonstandard scrollbars that worked — notably on Tiffany's site, which is so simplified in appearance that users couldn't miss the scroll controls even though they were fairly small and violated GUI recommendations (these deviations caused other usability problems, but at least people used the scrollbar). In general, users often overlooked nonstandard scrolling controls, and thus never got to scroll the lists to see the hidden options.
Design Guidelines Beyond Web Usability
The good news is that the tyranny of the browser has come to an end. We no longer have to squeeze functionality and feature-oriented design into a frame that was optimized for navigating hypertext and reading articles. The bad news is that the move to Internet-based applications requires designers to pay attention to a new set of guidelines, besides those we've identified for Web design.
In essence, we can state much of Web usability as: answer customers' questions, get to the point, and take it easy on the bells and whistles. If only we could get websites to stop annoying users, much would be gained.
Applications have to go much further than simply answering questions. They need features — the right ones, and not too many — presented in ways that empower users. This is a much tougher design challenge than simply providing information. Flash applications require a new level of usability commensurate with their status as ephemeral applications. A focus on simplicity will be key, as will a much deeper understanding of users' needs than has characterized the first decade of Web design.