Two Sigma: Usability and Six Sigma Quality Assurance

by Jakob Nielsen on November 24, 2003

Summary: On average across many test tasks, users fail 35% of the time when using websites. This is 100,000 times worse than Six Sigma's requirement, but Web usability can still benefit from a Six Sigma quality approach.

Do the management buzzwords "six sigma" relate to usability? Yes, but applying six sigma quality measures and techniques entails challenges: current Web usability is 100,000 times worse than the six sigma quality level. Also, user interface designers rarely employ systematic quality engineering methods. We have much to learn from manufacturing companies' more disciplined approach to quality.

"Six sigma" holds that any process outcomes that are within six standard deviations from the mean must be acceptable. (The Greek letter sigma, σ, is the traditional symbol for standard deviation in statistics.)

The following table shows quality measurements across a range of our recent projects, which tested usability on a total of 139 websites. The table shows the average user success rate, which I define as users' ability to accomplish their tasks.

User tasks Success rate
Using Web-based applications 45%
Shopping on e-commerce sites 56%
Finding company locations 63%
Using "About Us" information 70%
Using the Investor Relations area 70%
Using the PR area 73%
Subscribing to email newsletters 78%
Average success rate 65%

Users tested on intranets had a success rate of 75%. Intranets tend to score higher than websites because employees get accustomed to their company's intranet and thus are more likely to overcome its usability problems. This does not, however, liberate intranet designers from the obligation to consider usability. First, it's unacceptable to have employees failing 25% of the time when they're trying to do their job. Second, intranet designers must consider quality metrics beyond simple success rates. Time-on-task is particularly important because the company is paying for employees' time as they slowly slug their way across the intranet.

Comparing Quality Levels

As the table shows, when public website users perform simple Internet tasks, they're successful two-thirds of the time on average. In other words, users fail 35% of the time. This corresponds to a 1.9 sigma quality level according to how most six sigma people calculate quality. Intranets are at 2.2 sigma.

(Statistical note: Although I'd place a 35% defect rate at a lower sigma level, six sigma practitioners adjust their numbers to allow for a supposed "sigma shift." I've accepted their view here and adjusted accordingly so that my numbers can be compared to those in the literature.)

Six sigma tolerates no more than 3.4 defects per million manufacturing opportunities; in contrast, the Web generates 350,000 defects per million interaction opportunities. The difference between the two quality levels is a factor of 100,000. So why does the Web continue to work? Because people are flexible and will try again if they fail (though they’ll usually do so on a different website).

Many people hope that the Web will move beyond its current primitive features and support much more advanced user goals. However, as the table shows, the more complex tasks (applications, shopping) have the lowest success rates, while the simplest tasks (such as signing up for an email newsletter) have the highest success. Clearly, we'll need substantial usability advances before a more advanced Web can become feasible.

Six Sigma Process Is Achievable

For every thousand lines of commercial software code they ship, programmers typically leave six bugs behind, which corresponds to four sigma. Software engineering is about halfway between Web usability and manufacturing in its sigma level.

Software quality is shabby, but usability quality — which is languishing at two sigma — is pathetic. We'd be wise to adapt some of the six sigma methodologies to aid our quest for improved Web quality.

Six sigma quality engineering relies on a five-step process called DMAIC, which stands for define, measure, analyze, improve, and control. By employing each of these steps, Web projects can move toward better and more systematic quality achievements.

  • Define. As the very first step, specify which usability attributes your customers value most. Which top tasks must be easy for users to perform? How fast should users be able to accomplish critical goals?
  • Measure. I usually advocate qualitative usability studies, because usability's main goal is to drive the design. For formal quality assurance, however, you must run quantitative studies to collect hard numbers that show how well or poorly your design scores on the usability criteria you defined above.
  • Analyze. Most likely, there will be a gap between the measurement results and the level of quality you desire. Analyze the test results to identify the root causes; this not only lets you make specific design changes, but also helps you determine why these usability flaws made it into your design in the first place.
  • Improve. Fix the design. Removing the flaws is an obvious step, but you should also fix your design process so that you introduce fewer flaws the next time you design something.
  • Control. Don't slack. You must continue to monitor the quality level of your user experience as you introduce more advanced task support, and as users' expectations for usability increase over time. Improve your design methods and retain accountability for each team member's contribution to overall quality.

The last bullet may be the most important. A key lesson from many other fields is that continuous quality improvement is the way to true excellence. That's a lucky break: Web usability is so far behind that there's no hope of reaching acceptable quality in a single leap. Continuous improvement is our only chance.

Update added 2015: Website usability has improved since this article was originally written in 2003. We are now at almost 3 sigma. Still bad, but better than 2 sigma.

(More in the full-day training course Measuring User Experience.)

Share this article: Twitter | LinkedIn | Google+ | Email