Summary

The report serves three purposes:

• It documents how we collected our data so that readers can assess whether they should believe our findings. The world is getting crowded with "usability" reports. The quality of the recommendations depends on the quality of the methodology and the analysis. To assess our methodology, read this report.
• It allows others to replicate our study.
• It also allows others to run new and different studies that are based on our methodology: this will be a great help in planning your own studies, even if you don't plan to be as ambitious as we were and test 226 sites.

The description of our test methodology will be valuable for companies that plan to run their own competitive studies. We recommend that you study a smaller number of competing sites than the huge number of sites we tested, but otherwise the methods we used should be very useful for collecting your own data.

The report contains the complete questionnaires we used to collect pre- and post-test information from users.

As a rare feature for a usability report, we discuss several test tasks that were tried in pilot testing but had to be rejected or modified for the final test. The quality of the tasks is one of the most important determinants for the validity of a usability study.

86 pages.

This document is a part of Nielsen Norman Group's series of design guidelines for e-commerce user experience. Save 58% by buying all 13 reports on a single CD-ROM.

Process Guidelines

Discussions about website design based on usability studies are increasingly common. More now than ever, website designers, usability professionals, and those responsible for the business aspects of retail websites recognize the value of usability data in building profitable sites.

The demand for information about what really works and what doesn't is so great that anyone with a usability lab and a little time can conduct a study and publish a report. We can expect to see not only an increase in the number of usability studies but also greater variations in their quality. Ultimately, the recommendations arising from any usability study can only be as good as the information on which they're based. The quality of this information depends on the methods used to collect the data.

Our goal in writing this Methodology report was to expose the inner workings of our e-commerce usability study: the hows and whys of our methods. We want to establish a standard against which other studies can be measured. Our hope is to discourage poor-quality imitators while helping our peers (and ourselves) produce work of even higher quality in the future.

Table of Contents

1. Study Overview
   • Summary of First Study
   • Summary of Second Study
     • Diary Study
     • Georgia Testing
     • Indiana Testing
     • New York Testing
     • Hong Kong Testing
     • London Testing
2. Users and Recruiting
   • First Study
     • Recruiting
   • Second Study
     • Usability Tests
     • Diary Study
     • User Profiles
3. Testing Logistics: Locations and Equipment
   • Study One
     • Test Facilities
     • Testing Set-Up
     • Personal Information and Privacy
   • Second Study
     • Test Facilities
     • Testing Set-Up
     • Personal Information and Privacy
4. Selecting Sites
   • First Study
   • Second Study
     • Eyetracking Component (New York)
     • Additional Locations (Georgia, Hong Kong, Indiana and London)
5. Sites Tested
   • First Study
   • Second Study
     • Eyetracking Study (New York)
     • Open-Ended Tasks
     • Additional Studies (Hong Kong, Indiana, and London)
6. About Tasks
   • First Study
   • Task Refinement
7. Second Study
   • Types of Tasks
   • Task Refinement
   • International Tasks
   • Other Task Considerations
8. Task List
   • First Study
   • Second Study - Eyetracking
     • Specific Item and Customer Service Tasks
     • Criteria Tasks
     • Purchase Tasks
     • Known Sites
     • Open-Ended (No site specified)
     • Additional Tasks
   • Second Study – Additional Locations
     • Georgia Testing
     • Hong Kong Testing
     • Indiana Tasks
     • London
9. Running the Usability Tests
   • Test Facilitation
   • Personal Data or Fake Identity
   • Quantifiable Measures
10. Diary Study Methodology
    • Users
    • Methodology
    • Running the Study
11. Lessons Learned
    • Eyetracking
    • Webcam
    • On the Desk
    • Consent, Incentives and Reimbursement
    • Tasks
      • Motivation
      • Task Independence
      • Cultural Issues
      • Real Purchases
      • Open-Ended Tasks
      • Providing Fake Information
      • Testing "Known" Sites
    • Diary Study
12. Study Materials – First Study
    • Pre-Test Questionnaire
    • Post Task Questionnaire
13. Study Materials – Second Study
    • Post-Task Questionnaire
    • Diary Study Welcome Letter