Measuring the Usability of Reading on the Web
's column on how users
read on the Web
We measured five usability metrics for each version of our website:
was the number of seconds it took users to find answers for specific questions about the content.
was a percentage score based on the number of incorrect answers users gave for questions that had a known answer (one question asked users to determine their favorite tourist attraction: there was no one correct answer, so this question was not scored for errors).
comprised two measures from an exam given to the users after they had finished using the site.
was a percentage score based on the number of correct answers minus the number of incorrect answers to 5 multiple-choice questions.
was a percentage score based on the number of items correctly recalled after the test minus the number incorrectly recalled (users were asked to list as many of the tourist attractions discussed in the site as they could remember).
Time to recall site structure
was the number of seconds it took users to draw a sitemap. This is a measure of how well the users had understood the information architecture: if they understood it well, they would draw it quickly; if they understood it poorly, they had to think longer.
was determined from participants' answers to a questionnaire. Each question used a 10-point rating scale. Four satisfaction criteria were averaged to derive the subjective satisfaction score: perceived
(e.g., "How satisfied are you with the site's quality of language?"), perceived
ease of use
(e.g., "How easy is it to find specific information in this website?"),
(e.g., "the term 'fun to use' describes the site very well"), and
(e.g., "How tired do you feel right now?").
Note that the subjective metric assessed how well users
the site worked, not how well the users actually performed. It was quite possible for a user to be very slow at answering the questions and still say that he or she thought that it was very easy to find information on the site.
Overall usability of a site was calculated as the geometric average of these five measures. Each measure was normalized relative to the performance measured for the control condition (for example, if users could remember 5 things in the control condition but 6 things in one of the other conditions, then that condition received a 120% score for memory).
In our study, we gave equal weight to each of the five usability metrics when computing overall usability. Depending on the goal of a project, it may be better to use different weights:
site might give added weight to the
measure and perhaps also some added weight to learning the
are highly performance oriented
and should enhance employee efficiency, an intranet project would give the highest weight to
(a site for customer service reps might place the highest weight on avoiding errors )
site would place the highest weight on
and might give zero weight to errors and very low weight to other performance metrics.
More on Metrics
See also my report on
usability metrics and return on investment (ROI)