When to Outsource the Recruiting of Test Users

by Jakob Nielsen on May 3, 1998

Sidebar to Jakob Nielsen 's column on the cost of user testing a website

In a comment on my claim that recruiting [of test users] can be outsourced to a focus group company (at a cost of less than $1,000 for five users), Elizabeth Buie wrote:

The remark about outsourcing user recruitment to a focus group firm does not apply to all projects. (I started to say it applies only to projects in which the users are representative of the general population, but then it occurred to me that a focus group company can certainly apply some kinds of selection criteria.) It would be especially difficult for an outside company to recruit from outside just the right sample of users for an intranet webspace of a highly specialized company or government agency. When your user population is relatively small and well defined, you have to recruit from those people, schedule their time, etc. In a recent analysis I did for such a government agency, I found that this took by far the largest amount of calendar time.

I completely agree with Buie's analysis. There are many usability projects that benefit from outsourced recruiting, but there are some that don't.

There are only three firm rules for usability testing :

  1. recruit representative users to serve as your test participants
  2. ask them to perform representative tasks with the interface
  3. shut up while the users work: you want to see what they do on their own initiative without giving them help or biasing their approach to the site - it's a usability test, not a demo

For an intranet usability test , the "representative users" are your own employees , so it doesn't make sense to ask an outside agency to recruit them. Instead, get your five test users by contacting random employees in five different departments or job categories. Depending on corporate culture, you may need to get the approval of the department managers before approaching their staff, but it is often best for you to initiate the contact and not have the manager do the recruiting. There will always be a temptation for a manager to ask a "model" employee to be the test user - either because the manager wants to show off the department in the best light or because the manager mistakenly believes that it is best to test the biggest Internet fan in the department. On the contrary, you obviously want as average an employee as possible for your test.

Similarly, sites that aim at a narrow group of external users may be better off doing their own recruiting. Sometimes, the users are a specific, well-defined group (for example, an extranet may aim at the purchasing agents at major customers). Other times, the site has a list of registered users or existing customers. In these cases, you will probably have an easier time contacting these potential test users yourself, and they will be more willing to listen to you than to an external agency.

For external websites that are aimed at the general population or at relatively broad groups (e.g., farmers or Perl programmers) it is easier to have users recruited by a company that specializes in recruiting. Your marketing department probably already has a contract with a focus group company that specializes in your industry and has all the necessary procedures in place to recruit people for studies. It can be quite time consuming to call around to potential users and try to build a suitable schedule. This job is often best outsourced so that you can concentrate on the design of your site.

Test User Compensation

Depending on the difficulty of finding people with the desired profile, the focus group company may charge between $50 and $100 per person. Furthermore, the test users typically receive between $50 and $100 each, so the total cost for five users is usually less than $1,000.

Certain kinds of studies can get away with paying their test users much less than $100:

  • sites aimed at children or students: such users typically get a nice toy or $10-15.
  • very specialized sites where an existing group of users have a vested interest in improving the site because they rely on it: people will often be happy with a T-shirt or other giveaway that proves their inside connection with the site

 

 


Share this article: Twitter | LinkedIn | Google+ | Email