An Extended Search Trek

by Jakob Nielsen on April 11, 2011

Sidebar to Jakob Nielsen's column on Incompetent Research Skills Curb Users' Problem Solving, April 2011

In a recent user study in Australia, we asked a user to find the most populous city in the world. The user proceeded through 22 pageviews as follows:

  1. Went to google.com.au
    • Searched for "city with greatest population"
  2. On the Google SERP (search engine results page), she reviewed the first 7 hits, reading the titles, summaries, and site names in the URL field.
    • She then selected hit #5, WorldAtlas.com, with a deep link from Google directly to a page titled: City Populations – World City Populations, Biggest Largest Cities. She skipped over the higher-ranked sites (CityMayors.com and Wikipedia), saying, "I'm not necessarily going to go into Wikipedia, although for something like this they would probably be fairly accurate."
  3. On WorldAtlas.com, she found an answer: Tokyo, with 32,450,000 inhabitants.
    • She was not completely confident that this was the correct answer, and said that she would "go to another couple of websites and just use that to cross-check."
  4. Returned to the Google SERP by clicking the Back button.
    • She selected hit #3, Wikipedia, with a deep link from Google directly to a page titled List of cities proper by population. As hit #4, the SERP listed a second Wikipedia page titled List of cities in Australia by population , but she didn't select this link because she wanted to find the world's largest city.
  5. On en.wikipedia.org, she found an answer: Shanghai, with 12,831,900 inhabitants.
    • Noticing the discrepancy with the previous site's answer, she hypothesized that the difference might be due to the date of the different statistics.
  6. Returned to the Google SERP by clicking the Back button.
    • She reselected the link where she found the first answer (hit #5, WorldAtlas.com).
  7. WorldAtlas.com stated that some data had not been updated in 2010, so she concluded that the Tokyo number might be from 2009 (the site didn't indicate the year).
  8. Returned to the Google SERP by clicking the Back button.
    • She selected a new site — hit #6, Mongabay.com — with a deep link from Google directly to a page titled World's Largest Cities [rank: 1-1000].
  9. On Mongabay.com, she found an answer: Tokyo, with 34,400,000 inhabitants.
    • She noted that "because so many websites feed from one another, they could both be feeding from the same dataset, so I need to keep looking."
  10. Returned to the Google SERP by clicking the Back button.
    • Selected hit #7 , PubQuizHelp.com, with a deep link from Google directly to a page titled Cities - Highest Population | Geography | Pub Quiz Help.
  11. On PubQuizHelp.com, she found an answer: Shanghai, with 14,608,512 inhabitants.
    • She stated that, "I would have to keep searching further, because I am not satisfied with the information that I've got as yet." (She also noted that it "could be forever on the Web" to find the correct answer.)
  12. Returned to the Google SERP by clicking the Back button.
    • Scrolled down and selected hit #8, NationsOnline.org, with a deep link from Google directly to a page titled The Most Populated Cities of the World. World Megacities - Nations...
  13. On NationsOnline.org, she found an answer: Shanghai, with 14,348,535 inhabitants.
    • She noted that this data was from 2007.
  14. Returned to the Google SERP by clicking the Back button.
    • She scrolled back up and reselected hit #3, Wikipedia.
  15. On en.wikipedia.org, she saw a footnote citing a 2007 database as the source, and concluded that its Shanghai number was from 2007. In fact, that footnote referred to other data in the Wikipedia article; the Shanghai population count was referenced from another source in another footnote as being from 2008.
    • She concluded, "I am starting to feel that the ones that are listing Shanghai are the 2007 figures and the ones that are listing Tokyo are probably 2009 figures, so slightly more accurate."
  16. Returned to the Google SERP by clicking the Back button.
    • She scrolled further down and selected hit #10, Geography.About.com, with a deep link from Google directly to a page titled Largest Cities in The World - List of the Largest Cities.
  17. On About.com, she found an answer: Tokyo-Yokohama, with 33,200,000 inhabitants.
    • She noted that this list was based on data from 2005, "so it's getting worse."
  18. Returned to the Google SERP by clicking the Back button.
    • Editing the query string by appending the term "2010 data" to the existing query, she then searched for "city with greatest population 2010 data."
    • The resulting search hits mainly listed country populations (plus, for some reason, the population of San Jose, California). She then said, "I'm not liking the search results I am getting there."
    • Selected hit #8, WholesomeWords.org, with a deep link from Google directly to a page titled Statistics Population, World, Countries, Cities, Religions, Roman…
  19. On WholesomeWords.org, she got a very long page with various statistics and scrolled down only about 20% of the page before abandoning the site. She didn't see the information she was looking for, even though it might have been further down the page.
  20. Returned to the (revised) Google SERP by clicking the Back button.
  21. Returned to the original Google SERP (without the added query terms) by clicking the Back button again.
    • Reselected hit #5, WorldAtlas.com.
  22. While on WorldAtlas.com, which was the first page she had visited, she concluded that the most likely answer was Tokyo, because "this seems to be the most recent data set."

It's interesting that this user never selected the first two Google hits, which is what people typically choose. These hits where both from CityMayors.com, which is, according to the site, "an international think tank dedicated to urban affairs." (The site's answer was Seoul, with 10.2 million inhabitants.)

Because the user never commented on these top hits, we can only speculate about why she didn't click them. Based on our observations, it seemed that she was initially attracted to hits in the middle of the page — from Wikipedia and Worldatlas.com — and upon revisiting the SERP she simply scanned down from there, as opposed to looking back up.

The user's task performance had two main problems:

  • She didn't realize why she was getting very different answers to a fairly straightforward question. The reason is that Tokyo is the world's largest city (approximately 33 million people) when considering entire metropolitan regions, including outlying areas such as Yokohama, while Shanghai is the largest (approximately 13 million people) when counting only the official city area itself.
  • Our user never went beyond the specific Google SERP list to track down an authoritative source that might explain how city populations are measured and why they're estimated in such widely differing ways.

Share this article: Twitter | LinkedIn | Google+ | Email