Hidden- and Visible-Navigation Study: Methodology

This article is a sidebar to ”Hamburger Menus and Hidden Navigation Hurt UX Metrics” article, which includes the research study findings.

The study was run in December 2015. We used the WhatUsersDo remote testing platform to recruit users, administer tasks, record sessions, and watch the resulting video recordings. WhatUsersDo researchers worked with us to tag video recordings and log metrics for each sessions. We checked the logs, and analyzed the data using the R statistical software.

Sites and Tasks

We used the mobile and desktop version of 6 different sites (listed below):

• 7digital (music, e-commerce)
• BBC (news)
• Bloomberg Business (business news)
• Business Insider UK (business news)
• Supermarket HQ (e-commerce)
• Slate (news)

For each site, we wrote 2 tasks:

• Task 1 could be completed without the use of general site navigation because it asked for something that was linked from the homepage. For example, on the 7digital website, task 1 read, “See if there are any new songs or CD's you are interested in at https://www.7digital.com. Once you find something, please say which one you choose.” This could be completed by selecting a link among the albums and songs featured on the front page at the time of the user’s visit. (This site always has some music offerings on the homepage.)
• Task 2 required the use of navigation (or search) because it involved something that was not available on the homepage. For example, on the BBC site, task 2 read, “Go to http://www.bbc.com and see if there is any information on nature or wildlife you are interested in. Once you find a particular article that is interesting, please say which one it is.” To do this, users had to expand the More navigation menu and select the Earth option (or they could use search).

The sites that we tested used a variety of hidden, visible, and combination navigation patterns.

Navigation type for the different sites used in our study (Note that BBC was a responsive site that delivered different layouts to different phone screen sizes. Thus, on some phones its navigation was completely hidden, whereas on others it was partially visible. We coded the two types of sessions differently.)

Participants

• 179 people participated in our study; 94 were male and 84 were female.
• 80 participants used the sites on their phones and 99 participants used the sites on a desktop. Nobody did the study both on desktop and on mobile. (Thus, the device type was a between-subjects independent variable.)
• All participants lived in the United Kingdom.
• UK socioeconomic demographic classification (NRS):
  • A (upper middle class): 7 participants
  • B (middle class): 110 participants
  • C1 (lower middle class): 50 participants
  • C2 (skilled working class): 6 participants
  • D (working class): 3 participants
  • E (those at the lowest of subsistence):  2 participants
  • Students: 1 participant
• Ages:

Procedure

Each participant completed 6 tasks: one on each of the 6 sites mentioned above. 3 of the 6 tasks completed by each participant required no use of the navigation (these 3 tasks were randomly selected). The order of tasks was randomized for each participant.

We did not ask users to think aloud in the studies to avoid inflating time measurements and to reduce data variability.

Findings and Other Articles Related to This Study

Hamburger Menus and Hidden Navigation Hurt UX Metrics.

Mobile First Is NOT Mobile Only

How to Make Navigation (Even Hamburger) Discoverable on Mobile

Beyond the Hamburger: How to Make Navigation Discoverable on Desktops

(Learn how to conduct your own UX measurement studies in our full-day training course.)