Cloze Test for Reading Comprehension

by Jakob Nielsen on February 28, 2011

Summary: Cloze Tests provide empirical evidence of how easy a text is to read and understand for a specified target audience. They thus measure reading comprehension, and not just a readability score.

Sidebar to Jakob Nielsen's column on Mobile Content Is Twice as Difficult, Feb. 2011.

It's easy to estimate the difficulty of your content: simply run the text through a readability formula. The resulting number is the reading level, usually stated as a grade level that corresponds to the number of years of formal education that are required to understand the text.

There are many different readability formulas, but most people use Flesch-Kincaid simply because it's built into Microsoft Word (under Review > Spelling&Grammar). For example, Flesch-Kincaid rates this article at an 11th-grade reading level, meaning that it can be read by high school seniors — people who've already completed 11 years of education.

Unless you're a readability expert, the differences between formulas don't matter much, so just use whatever is close at hand. When we assess a website's copy, it doesn't matter whether it computes at, say, 11.3 or 11.6. In either case, that's a high school reading level, meaning that it's too difficult for a mainstream site but acceptable for a site targeting business professionals. (When targeting a broad consumer audience, you should write at an 8th-grade reading level.)

Readability vs. Comprehension

Readability rates the text's complexity in terms of words and grammar, but we're actually more interested in the text's difficulty in terms of reader comprehension of the content. Sad to say, no formula can measure whether users understand your site.

Take, for example, the following two sentences:

  1. He waved his hands.
  2. He waived his rights.

Both score well in readability formulas: simple words, short sentences. But whereas everybody understands what the first sentence describes, you might need a law degree to fully comprehend the implications of the second sentence.

(This hands/rights example comes from the paper by Singh et al. discussed in my main column.)

In addition to pure literacy skills, comprehension depends on a mix of IQ, education, and background knowledge. Thus, to measure comprehension, you must test with real users from your target audience.

> Read more in the separate article Legibility, Readability, and Comprehension: Making Users Read Your Words.

Cloze Test Procedure

The Cloze test is a common empirical comprehension test. It works as follows:

  1. Replace every Nth word in the text with blanks. A typical test uses N = 6, but you can make the test easier by using a higher N value.
  2. Ask your test participants to read the modified text and fill in the blanks with their best guesses as to the missing words. Each person should work alone.
  3. The score is the percentage of correctly guessed words. Because you're testing comprehension rather than spelling skills, synonyms and misspellings are allowed.

If users get 60% or more right on average, you can assume the text is reasonably comprehensible for the specified user profile employed to recruit test participants. There's a clear difference between readability scores and comprehension scores:

  • Readability is a property of the text itself and predicts the education level typically needed for people to read the content without undue difficulty.
  • Comprehension is a combined property of the text and a specific user segment and indicates whether this target audience actually understands the material's meaning.

Here's an example, using a paragraph from Facebook's privacy policy:

Site activity information. We keep {1}______ of some of the actions {2}______ take on Facebook, such as {3}______ connections (including joining a group {4}______ adding a friend), creating a {5}______ album, sending a gift, poking {6}______ user, indicating you “like” a {7}______, attending an event, or connecting {8}______ an application. In some cases {9}______ are also taking an action {10}______ you provide information or content {11}______ us. For example, if you {12}______ a video, in addition to {13}______ the actual content you uploaded, {14}______ might log the fact that {15}______ shared it.

(Solution at the bottom of this page.)

The full text — before inserting the blanks — scored at a 14th -grade reading level, corresponding to having completed 2 years of university. Thus, if you're a typical smart Alertbox reader, you can probably understand the paragraph and complete the Cloze test. Still, this is a higher reading level than what's required for much of the younger Facebook audience. Most teenage users need far easier text, and even college students prefer non-college level text when they're online — leisure sites shouldn't feel like textbooks.

Solution to Sample Cloze Test

Don't peek, if you want a go at solving the Cloze test on your own. Here are the missing words from the sample paragraph:

{1} track
{2} you
{3} adding
{4} or
{5} photo
{6} another
{7} post
{8} with
{9} you
{10} when
{11} to
{12} share
{13} storing
{14} we
{15} you

Did you get at least 9 of these right (corresponding to 60%)? If so, you can probably comprehend the full text fairly easily. If you got a lower score, that doesn't prove that you're stupid or that the text is densely written. The problem is likely to be a lack of contextual knowledge of Facebook. For example, the word "poking" is generally easy enough to understand, but its meaning in the Facebook privacy policy context is completely incomprehensible unless you're a user. (Which is okay, because any given text needs to be comprehensible only to the target audience.)

Full Report

Full eyetracking report on how users read on the web is available for download.

Share this article: Twitter | LinkedIn | Google+ | Email