Summary

The report serves three purposes:

• It documents how we collected our data so that readers can assess whether they should believe our findings. The world is getting crowded with "usability" reports. The quality of the recommendations depends on the quality of the methodology and the analysis. To assess our methodology, read this report. To assess our analytical skills, review the experience of our team.
• It allows others to replicate our study.
• It also allows others to run new and different studies that are based on our methodology: this will be a great help in planning your own studies, even if you don't plan to be as ambitious as we were and test 20 sites.

The description of our test methodology will be valuable for companies that plan to run their own competitive studies. Normally, we recommend that you study a smaller number of competing sites (20 is too much except if you want to write a fancy report series), but otherwise the methods we used should be very useful for collecting your own data.

The report contains a set of forms that can be used for running your own studies:

• Screening questionnaire for recruiting test users
• Consent form
• Pre-test questionnaire
• Post-test questionnaire

As a rare feature for a usability report, we discuss several test tasks that were tried in pilot testing but had to be rejected or modified for the final test. The quality of the tasks is one of the most important determinants for the validity of a usability study.

48 pages.

This document is a part of Nielsen Norman Group's series of design guidelines for e-commerce user experience.
 

Process Guidelines

Discussions about website design based on usability studies are increasingly common. More now than ever, website designers, usability professionals, and those responsible for the business aspects of retail websites recognize the value of usability data in building profitable sites.

The demand for information about what really works and what doesn't is so great that anyone with a usability lab and a little time can conduct a study and publish a report. We can expect to see not only an increase in the number of usability studies but also greater variations in their quality. Ultimately, the recommendations arising from any usability study can only be as good as the information on which they're based. The quality of this information depends on the methods used to collect the data.

Our goal in writing this Methodology report was to expose the inner workings of our e-commerce usability study: the hows and whys of our methods. We want to establish a standard against which other studies can be measured. Our hope is to discourage poor-quality imitators while helping our peers (and ourselves) produce work of even higher quality in the future.

Although all of us have extensive experience in planning and conducting website usability studies, this project was especially challenging because of the large number of sites, short time frame (we completed all testing within a month), and the geographic separation of the five primary team members.
 

Table of Contents

1. Introduction
2. Sites We Tested
3. Type of Study
4. Our Team
5. Study Overview
   • What Is Usability Testing?
   • Common Misconceptions about Usability Testing
   • Number of Users
   • Number of Sites
   • Test Length and Sites per Test
   • Which Users Tested Which Sites
   • Site Order
   • Timing Concerns
   • Site Selection Criteria
6. Users
   • User Profiles
   • Demographics of the Study Participants
   • User Recruitment
7. Creating Good Tasks
   • Realistic Scenarios
   • Motivation
   • Task Independence
   • Cultural Issues
   • Task Descriptions
   • Tips for Writing Task Instructions
   • Task Refinement
   • Testing of Complete Purchases
8. Usability Test Facilities
9. Test System Configuration
   • Hardware
   • Software
   • Between-Test Cleanup
10. Pilot Testing
    • What We Changed
11. Test Facilitation
    • Description of Think-Aloud Protocol
    • Personal Data or Fake Identity
    • Assisting with Technical Problems
12. A Discussion of Bias
13. Data Collection
    • Post-Test Questionnaires
    • Data Analysis
14. Defining Success and Failure
    • Task Failures
    • Sales Catastrophes
    • Examples of Task Failures and Sales Catastrophes
15. Testing Websites: A Cautionary Tale
16. Participant Screening Questionnaire
17. Consent Form (USA)
    • Introduction
    • Statement of Informed Consent
18. Pre-Test Questionnaire
19. Site Feedback Questionnaire
20. Tasks