These are the instructions we gave to the people at various Sun branch offices in Europe and Asia for their user testing of a new design for the company's web pages. In a few places, these instructions refer to web-specific issues, so they will have to be modified slightly for use in other projects. These instructions were sent by electronic mail to those local Sun reps who had volunteered to lead a test.
Thank you again for volunteering to help perform international usability testing on Sun's new WWW pages. A prototype of the redesigned pages will be available on the following URL: http://foo.bar/sun.categories/
Some info is there now, but the graphics will not be available until Monday, May 1, 1995. You can try out the information structure now and you should schedule your user for as soon as possible after May 1.
Step 1. Getting a Customer
Using your contacts to customers in your location, call around and recruit a person who is a typical user for the kind of accounts you expect to be most important in the future. The user should be average for WWW users in that organization and not a super expert or high-level manager (unless they are the only ones using the WWW). The user should have some ability to read written English since the pages will not be translated, but the user should not be uncommonly good at English (and will not need any skills in spoken English). In recruiting the user, you should explicitly ask how much he or she has used the WWW (or use terms like "Mosaic" or "Netscape" if they don't know what WWW means). For the WWW test, the user should have had previous experience with the WWW, though he or she need not have visited Sun's home page. Since you will be accessing files inside Sun's internal network, you will need to ask the user to come to your office. Set up an appointment for a specific date and time and tell the user that the test will last about an hour but might go a little bit longer or might be over faster, depending on what happens.
Tell the user that volunteering for the test will help Sun make a better user interface for people in your country and that the results of the test will be communicated back to headquarters without revealing the identity of the user or his/her company. That is, all results will be kept confidential.
Step 2. Preparing the Test
Make sure that you have a room with a workstation reserved for the test and that you will have no interruptions during the test: shortly before the test you should put a sign on the door reading something like "user testing, do not enter" and unplug the telephone. Also, shortly before the test you should make a last check that you have your WWW software (e.g., Mosaic) running so that there are no surprises once the user gets there. The last part of setting up the test is to make the browser software forget the navigation history so that all link anchors are blue (usually, the ones the user has already seen are purple).
At least a day before the test you should translate the test tasks into your local language (the test tasks are described below). Go through the test tasks yourself to make sure they can be done and that the instructions make sense. While doing so, you may observe some aspects of the user interface that seem poorly suited to users from your country, and you should make a note of any such problems.
Step 3. Running the Test
When the user arrives, explain once again that you are doing a usability study to see how easy it is to use Sun's new WWW pages. Explain that you are going to be trying a prototype and that not everything has been completed yet, meaning that errors may occur during the test.
Ask the user to "think out loud" during the test: the user should keep up a running monologue, saying anything that comes to mind and not hold back anything. Say that you are interested in all the user's thoughts, comments, and interpretations of the design, even the ones that may seem minor at the time. Since thinking aloud is somewhat unnatural, many users stop doing so and need to get prompted to speak. If the user sits quietly for an extended period of time, a good prompt is "what are you thinking about now?"
Other than prompting the user to keep talking, you should not say anything yourself during the test. If the user asks about the meaning of some element of the user interface, you should not answer, but instead ask the user "what do you think it might be?" Encourage the user to make a wild guess if necessary. Also, you should neither agree nor disagree with the user's comments. Just make a sound to confirm that you have heard the user's comment without indicating your personal opinion and then write down the user's comment. Actually, you can't write down everything the user says, but try to make as extensive notes as possible as that will help you write the report.
Step 4. The Test Tasks
There are two tasks: exploratory and directed. Of the one-hour test time, you should spend about 30 minutes on exploratory tasks, about 20 minutes on directed tasks, and about 10 minutes on debriefing (see the next section).
Bring up the new home page and ask the user to begin by giving his or her general comments on the page before starting to click. When the user has finished commenting on the page, you tell the user to do anything he or she would normally want to do on that page. You just let the user navigate the system freely for the rest of the exploratory test time. You may need to help the user get back to the home page if the user gets hopelessly lost, but don't offer help too soon since it is often very valuable to see how users get out of trouble on their own.
At the end of the exploratory time, you make sure that the user has seen and commented on the following new features in the system:
- The "What's Happening" page
- The news story about network security
- The column (under What's Happening) called User Interface Alert Box
- The "About this server" page
It is likely that the user will have visited some of these pages during the free exploration, and if so, you need not return to them.
Finally, give the user these tasks (translated):
- "You have heard that Sun recently introduced a new, low-cost desktop workstation. Can you find the product specification information for it in the Web site?"
- "You have recently acquired some Sun systems, and you are now looking for a good word processor to run on them. Can you find what word processing programs are available for the Sun platform?"
These tasks should be given in writing. Give the test tasks to the user one page at a time (you want them to concentrate on one task at a time). If you come to the end of the scheduled test time or if the user starts feeling uncomfortable, you should stop the test and not give out any remaining test tasks.
Step 5. Debriefing the User
After the test tasks (and while sitting at the computer), you tell the user that this completes the test. You should then ask the user for his or her general comments about the design. Use two questions: "What did you particularly like about this system?" and "What did you particularly dislike about this system?" If the user does not have enough comments, you can often elicit additional comments by asking the question "If we now compare with other sites you have visited on the Web, what things have you seen that you liked and would want us to do, and what things do you want to warn us against doing?"
After the test, you should thank the user for helping and mention how you have made a lot of interesting notes that you will now send back to the development team at Sun's headquarters. If doing so is appropriate for your culture, you should also give the user a gift to thank him or her for participating. You can use the Multimedia and Hypertext book we sent you for this purpose.
Step 6. Reporting the Results
Please respect the anonymity of the user and his/her company and do not mention them in the report. We only need to know the general type of work done by the user and the general type of company.
In writing the report, you should refer to both usability problems experienced by the test participant and to issues you observed yourself while preparing for the test. Please distinguish between these two kinds of data: we want to know what issues are your personal critiques of the use interface and what issues refer to actual events during the usability test. Both are valuable information, but we should treat them differently. When reporting data from the user, please also distinguish between cases where the user said that something might be confusing or problematic to other people in your country and where that specific user actually was confused or had problems.
As a guideline, a typical report would be maybe two pages long, but of course you should write as much or as little as needed to describe the major findings of your specific study.
Please email the report to me at email@example.com as soon as possible since we are on a tight deadline to finish the redesign effort.
For Further Reference
Read the following sections in Jakob Nielsen: "Usability Engineering" (paperback edition: AP Professional, ISBN 0-12-518406-9):
- 6.4: Ethical aspects of tests with human subjects
- 6.6: Stages of a test
- 6.8: Thinking aloud
For a discussion of the types of usability problems you might find and good ways of conceptualizing them according to established usability principles, read Chapter 5: Usability Heuristics