Group Notetaking for User Research

Introduction

Teams should collaborate on user experience activities in order to be sure to make products that are usable and desirable. Collaboration and team ownership of the user experience has many other benefits, in particular saving time by helping people design more-usable software to begin with, once everyone understands the users, their goals, and difficulties. A very effective way to get people involved is to invite them to observe user-testing sessions. Observing can be a great introduction to the value of user research, and gathering data together can bring benefits to all concerned, including:

• Gaining insights and developing high-quality solutions, as observers interpret incidents through their diverse professional points of view
• Engaging the team constructively, as everyone understands usability problems together
• Generating top findings together to get consensus quickly and motivate the team to fix problems
• Demonstrating how fast you can gather actionable information through UX research

A practical number of notetakers is 4 to 10 per session including designers, researchers, technical leads, and other stakeholders. Decide who needs to watch most or all of the sessions, then rotate a few other stakeholders into available slots. Observers should watch at least 2 sessions in a study, so they won’t be tempted to form judgments based on 1 user.

During user-testing sessions, most researchers take notes alone or in pairs, but when more people are involved, a bit more structure is necessary. In most UX research situations, it’s best to use the simplest tools for the job, and reduce reliance (and load) on the network. You can take group notes in a word processor, in a spreadsheet, on sticky notes, or on colored paper.

Each person should take notes separately. Later, the group or the researcher can combine and sort the observations. Shared collaborative documents can work fine for a couple of people, but for larger groups, shared documents can distract from the main task of observing, becoming unwieldy when everyone tries typing at once. Worse, seeing others’ notes can lead to groupthink or can diffuse responsibility and discourage people from taking notes, because someone else is already doing it.

Group notetaking involves 2 steps:

1. Capturing notes: each individual writes down observations during the session.
2. Collecting insights and observed problems: after each session, the main findings are summarized on a whiteboard. This is a group activity.

Techniques for Capturing Notes

There are 2 main ways of recording user data: chronological logs and topical notes. Both methods work well, so choose one based on your situation, or combine them.  

Chronological Logs

Chronological logs are lists of observations made during a user test, in the order in which they occurred. Typically, notetakers use a document template that includes the facilitator script (with user scenarios) and provides space to take notes. Each notetaker uses 1 copy of the document per participant.

Below is an example of a chronological log for a device-based study. This template could be presented in a word processor or as a spreadsheet. The facilitator reads aloud scenarios with the bold words or hands the user these scenarios one at a time, printed on paper (or in remote studies, by pasting them into chat or SMS).

Notetaking rows contain the participant’s number (P1, P2, etc. — no names), the scenario and task identifiers, the note, categorization of the observation, such as topic (Category column in our example), and importance. Researchers typically invent their own abbreviations for quick logging. Some of the information can be added before or after the session (for example, the first 2 and the last 2 columns). Some fancy systems and spreadsheets can add timestamps, or you can just manually note clock times for key quotes or events that you want to review later in recordings. Your own comments, suggestions, or insights about the interface should be marked as such and be distinguished from user-based observations. (The example above uses “me.”) A separate Comments column can sometimes be helpful.

Two quick shorthand notations can be entered into the comments or Category column when appropriate:

• V for Video: this segment might make for a compelling video clip to include in a highlights reel after the study.
• M for Methodology: we may want to revise our study methodology to handle situations like this better in the future. For example, the facilitator might have asked a question in a manner that biased the user’s response.

In theory, you could provide paper copies of the notetaking template, but in practice it makes little sense to do so. As a result, chronological logs are used mostly in digital format — each notetaker records observations on a laptop or another portable device. 

Advantages of digital chronological logs:

• Observers can be in several locations.
• Handwriting legibility doesn’t matter.
• Time data and sequences of actions can be compared or recreated, and click paths can be used to analyze navigation problems and solutions.
• You can combine notes from all the observers and then sort the rows in useful ways.
• Notes are immediately searchable.
• Digital raw data can be stored with your project plan.

Disadvantages of digital chronological logs:

• Everyone has a distraction machine in front of them.
• Facilitators may have to travel with 2 laptops (1 for taking notes, 1 for the user) when testing away from the office.
• People have to type very fast during the test sessions, which may be fatiguing as well as noisy.
• You have to categorize, rate, and count everything afterward in long, scrolling, digital documents.
• When people type, they tend to write more, which can cause extra data-analysis time if paragraphs contain many issues.

Tips: Remind everyone to use session time to capture as many observations as possible rather than to write neat stories. During data analysis, you can add more columns to put additional categories in, or you can import log data into a database or mindmap.

Topical Notes

If your team is in one location, using square sticky notes or small notepaper for group notetaking can be a great choice. With this method, each observer writes 1 incident per note. When doing qualitative studies with a small sample size, each participant is represented by a different paper color. Each note also includes the scenario letter and task number and the notetaker’s initials, with optional note topic.

Some observers can be asked to focus on capturing specific types of information, such as quotes, search queries, or click paths. After the research sessions end, the group sorts notes into emergent topical categories and prioritizes them through voting.

This method is commonly called affinity diagramming or KJ Analysis.

Advantages of topical notes:

• This method is easy to explain and to learn.
• Some people prefer to take notes on paper rather than typing.
• Observers don’t need  laptops.
• Paper is less distracting than internet-connected devices can be, so people stay focused on observing.
• There’s no typing noise to distract participants or observers.
• Using small notes encourages people to change paper when they change topics, making it easy to sort the notes later.
• Hand writing allows people to sketch and diagram if they want to.
• The group can sort notes after the study, shortening the work of analyzing the data.
• Color-coding makes it easy to see at a glance how many users had which kinds of issues.
• Sorting notes into topics immediately shows which issues generated the most notes.
• It’s easy to get team agreement quickly on a top-findings list right after the test. If a report is not necessary, then sometimes that’s all you need to move forward.

Disadvantages of topical notes:

• Hand-written notes can be difficult to read.
• Although you gain in categorization, you can lose in sequence.
• The process may appear to be less scientific to some stakeholders.
• Important data must be digitized or transcribed later; it’s harder to document the process or the resulting artifacts.
• A lot of paper is wasted.

Tips: Assign one of the observers the duty of collecting all the notes and distributing the next color to all the notetakers after every session. Sticky notes are good for grouping into topical categories on walls or windows. Small index cards or nonsticky paper notes are easy to sort on big tables.

Two examples of topical notes: The top contains the notetaker’s initials, the scenario letter and task number, and maybe a topic. (Topics can be added at any time, or notes can just be sorted into categories later.) Notes typically contain observations, questions, quotes, or remarks and ideas from the observer. Colors represent particular participants. (The reason we use color-coding to differentiate participants instead of notetakers is that it’s more important to see at a glance how many users encountered a certain usability problem than how many observers spotted that issue.)

Collecting Insights: Whiteboard Top-Findings List

Regardless of your chosen notetaking method (logging or topical), after each session schedule a short team-debriefing period to gather key findings quickly. Ask observers to describe the most important things they noticed during the preceding session. Capture those key observations in view somewhere, on a whiteboard or on a series of giant sticky notes, for example (but make sure that participants cannot see them).

The facilitator can field questions, provide insights and context, and help observers think constructively and objectively about the findings. Keep a tally next to items that are found again in additional sessions.

At the end of the research study (or anytime you run out of room), photograph the preliminary top findings for the record, because sticky notes tend to fall off the wall overnight, and cleaning staff may erase your whiteboard.

After the notes have been analyzed, the details and variations of the top findings, along with issues that weren’t raised in the debriefings, can be added to help complete the picture.

Advantages of the running top-issues list:

• Important findings stay visible and top of mind during the research study.
• Stakeholders can get up to speed quickly just by walking into the room and reading the board.
• Preliminary top findings can be transcribed, sorted, and conveyed to the larger group almost immediately after the study ends. For this reason, the whiteboard method is particularly suited for fast-paced Agile environments.
• Everybody collaborates on making the list, so the findings aren’t controversial or mysterious. At the end of the study, everyone knows what happened and what’s most important to focus on in the next design iteration.

Disadvantages of the top-issues list:

• Writing on a board is erasable and can be difficult to read.
• The top findings must be transcribed.
• It’s common to run out of room and have to photograph the board, erase, and continue.
• Issues on whiteboards can’t be searched or sorted, so finding an existing issue to add a tally number to it can take more time.

Tips: The UX facilitator should write on the board, because top issues need usability-oriented phrasing, and the process requires a leader to keep the team focused and to answer questions. Ask 1 of the observers to transcribe the board into a spreadsheet or mindmap throughout each day, in order to make the information digitally usable as soon as possible. It’s tempting to make 1 top-findings list per participant, but that makes it more difficult to tally issues that recur over the study. Rely on the notes to separate findings by participant, instead.

Choose the most-useful method for engaging observers in a given project, considering the people you have and the data you need. It’s common for UX researchers to log data while observers write notes by hand, but it’s best when all observers stick to one method in order to simplify data analysis. One person with many two-sided pages of notebook longhand can slow you down considerably when everyone else is producing sortable notes. The whiteboard method should be used whenever debriefing occurs, regardless of the primary data-capturing method.

Conclusion

Taking notes is a great way to bring teams together to solve usability problems. As stakeholders learn more about users and UX methods by participating, UX researchers can spend more time generating actionable data and less time explaining. You can’t just throw people into a research situation unprepared, however. Here are some guidelines for observers with notetaking instructions that you can modify for your situation.