Selecting an Online Tool for Unmoderated Remote User Testing

by Nielsen Norman Group on June 1, 2014

Summary: A good automated usability-testing tool offers recruiting flexibility, the ability to write your own tasks, and a video recording of the session.


Let’s say you have just decided that unmoderated remote user testing is perfect for your next project. Perhaps because the scope of your project is limited to a specific feature, your participants are dispersed, or you need results fast. The second question (after deciding on the study methodology) is what tool to employ. For in-person user research pencil and paper might suffice, but remote studies require technology — usually in the form of a cloud service.

Unmoderated sessions require a more complex platform than moderated remote sessions, for which online-meeting software such as GoToMeeting is enough. For unmoderated user testing, you need an application that plays the role of a session facilitator: it guides the participants through the session and records what happens.

Thankfully there are many online tools available, but how do you pick one?

Keep in mind that the essence of qualitative usability testing is to gather behavioral insights, by watching people interact with an interface. It is not enough, as some online usability-testing services offer, to ask users questions about their experience after they’ve used a website.

Other tools are focused on gathering quantitative data by tracking where the user clicks. These are great for informing the information architecture or benchmarking the overall usability, but are less useful for iterative design.

The following is a list of basic requirements for selecting an online tool for qualitative research.

Display and Audio Recording

Since users’ verbal and written comments can be misleading, you have to watch their behavior on a website in order to understand what works or doesn’t. While conducting live studies you can follow along with the participants as they conduct activities, but during unmoderated studies you need to be able to watch a recording afterwards.

Screen recordings are helpful, but, in the absence of an audio recording, it is easy to miss why certain behavior occurred. If nothing is happening on the screen, is it because the participant is reading, or is she thinking about where to click next? The audio recording of participants verbalizing their actions is essential.

The ability to video-record the participant via webcam allows you to observe participants’ facial expressions and potentially extract supplementary information about how they feel while they are doing the task. It is a nice feature, but it is not absolutely necessary. It can be more difficult to recruit participants who have a webcam and are willing to be recorded, so make it a secondary requirement.

In summary:

  • Screen and audio recordings are a must-have.
  • Facial video from webcam is a nice-to-have.

Participant Panel or Your Own Recruiting

There’s no point in watching people use your system if they don’t match your target audience. Make sure you have some control over who participates in the test either through screening questions or by recruiting your own participants.

Some tools only offer unmoderated user testing with participants from their panel, while others will provide you with a URL to distribute to your own pool of testers.

Using a provided participant panel is fast and easy (especially if you recruit from a broad and general population): people are familiar with the study technology and methodology and can participate in your test as soon as they have time. Make sure you have the option to specify basic demographics as well as additional qualifiers such as “Must have a Facebook account” or “Must buy something online at least 1x/month.”

The downside of using panel participants is that many do these studies so frequently that they’ve learned to focus on certain aspects of the design and look for things to critique. To compensate for possible “professional testers”, recruit extra participants and exclude recordings from people who didn’t seem honestly engaged with the activities.

Being online means that you can test users on the other side of the globe as easily as people on the other side of the street. Depending on whether your site targets international customers, the ability to test worldwide may or may not be important, but it’s one more thing to consider. For foreign users, an additional consideration is how to get their comments translated into your language. All possible, but currently not well supported.

In summary:

  • B2B sites, sites that target elite or rich customers, and other services with narrow target audiences usually can’t use panels and must recruit the test participants themselves.
  • B2C sites and other services that target a broad audience usually get faster and cheaper results by using a panel.
  • Panels can be limited in their geographic coverage.

Write Your Own Set of Tasks

The activities that you want the participants to conduct have to be even more carefully written than the tasks for moderated sessions. The participant doesn’t have the ability to ask for clarification if they don’t understand the instructions and you can’t ask them to try again if they do the wrong thing. The tasks should also allow users to determine when they have reached an answer and when they should stop.

If users misinterpret your instructions and perform the wrong task, your test is wasted.

You’ll find that most tools provide sample tasks, which can be good suggestions, but don’t blindly copy them. The tasks that you give participants to do on your site or application are highly specific to your situation. Generic tasks, such as “Find out what this company offers”, are unlikely to give you good insights: to really delve into the usability of your system you will need to write your own tasks.

You also don’t want to be limited in the number of tasks that the tool lets you assign. Although many tools impose a limit on the session duration, you should be able to have as many tasks as you want during that time. If the session duration is 30 minutes, you should be able to have ten 3-minutes tasks or two 15-minutes one.

In our experience training other companies to run their own remote usability tests, writing tasks is where most people fail in getting the results they need from their studies.

Can Test Anything: Mobile, Desktop, Prototype, Website, App

Unmoderated remote user testing is not just for desktop websites. You can test mobile phones, tablets, and apps. The actual setup for testing mobile devices varies so be sure to look at the specifics for each service.

Unmoderated user testing can also work for testing prototypes. As long as you can upload your sketch to a server so you can direct participants to it via a URL, you should be able to test it with any unmoderated remote user testing tool. However, when testing prototypes it is often better to be able to interact with the test participants since they will often encounter non-working functionality or partly-written content. In these cases, careful questioning and guiding can elicit addional information while helping the user overcome a rough spot in the prototype. Therefore, a moderated remote method is often better for protypype studies.

In summary:

  • Testing desktop, mobile, and tablet designs: unmoderated remote testing is great
  • Testing with rare or emerging platforms: the remote testing services usually don't support these technologies
  • Testing prototype designs: having a study facilitator is usually best, so moderated remote methods are often better than unmoderated ones

Same-Day Results

To conduct a moderated study with 5 participants you need at least a full day, while 5 unmoderated remote sessions can be done in a matter of hours, since they can run in parallel. In practice, your sessions won’t all run at exactly the same time, but many tools offer fast turnaround. You’re likely to be able to watch at least one session within an hour after posting your study!

Since the point of conducting these sessions is to get feedback quickly so design changes can be made, you don’t want to have to wait days for the results to come in. Of course, if you have very strict screening criteria, even an unmoderated study might take longer.

Ok, So Which Tool Should You Choose?

The functionality and pricing changes often for many of the unmoderated testing tools, so be sure to compare them to your needs at the time of your study. Here are several tools that, at the time of writing this article, claim to meet the above-mentioned requirements:

Maybe you were hoping for a single recommendation, but as with many user experience questions there is no single answer. But be happy that you have several good services available to choose from — this was not the case in the past. The plethora of choices means that you can now be more critical in selecting a tool that suits the methodological requirements of your project.

Summary

When evaluating online platforms for automated remote user testing, choose one that provides you with a recording of the participant’s display and audio as they conduct tasks. You also want to have some control over the recruiting so you can ensure that the participants match your target audience.

More on remote user testing and the relation between this method and in-person testing in the full-day Usability Testing training course, which includes hands-on details on writing tasks, facilitating sessions, and more.


Share this article: Twitter | LinkedIn | Google+ | Email