Sometimes referred to as mobile, digital, or online ethnography, diary studies involve a longitudinal collection of responses from participants and are meant to “capture life as it’s lived.” Diary studies are a type of context method. This article illustrates three common tradeoffs in qualitative diary studies that tend to complicate the diary-study design.
When a Diary Study Is Appropriate
Before you ask participants to log their behavior, you should ask the following questions (appropriate for any research effort):
- What is your research goal?
- Is a diary study the best method for that goal?
There are several situations where a diary study may be appropriate:
- When you need to report on a habit or repetitive behavior (e.g., exercise)
- If the activity of interest is likely to be split across many different sessions (e.g., planning a vacation, purchasing a home)
- When you know the approximate time frame but not the exact moments of certain activities of interest
Tradeoff 1: Natural Behavior vs. Incentives
Often, in diary studies, participants are paid per entry to sustain engagement and to adequately compensate them for their time and effort. There is usually a minimum number of entries that need to be submitted for some baseline payment and then a bonus amount for each additional entry. However, if you are paying per submission, participants may overproduce responses in order to maximize their monetary reward.
Suppose you ask participants in a 3-week diary study to report behaviors related to tasks such as vacation planning. You can pay participants weekly and set a lower limit for the number of entries per week. For example, respondents should provide at least five entries during a week to receive the $90 compensation for that week. To incentivize participants to submit more information during any one week, you may compensate with $5 per entry for up to a maximum of $100 total compensation per week. The maximum amount per week will keep people from making up entries to increase their overall gains.
Setting low and high incentive limits is important: it avoids the situation where one participant submits an overwhelming number of responses in one week while the other barely sends you a few entries.
The incentive amount per entry is also important. Too big of an incentive, and you may unintentionally lure people into creating more submissions than they would normally do to fit into the study. If the incentive is too small, people may quickly drop out of the study and decide it’s not worth their time.
It is exciting to see participants who are enthusiastic and eager to share their thoughts. However, you do not want to undermine the study by creating a diary-study response factory where people want to hit that magic number of entries to get paid. You can emphasize response quality in the initial interview with participants and encourage regular reporting with bonuses.
Continuously monitor the number and quality of responses as they are being submitted. Send followup messages and reminders to keep participants on track.
To determine the total incentive, consider the following:
- The complexity of each diary entry: do people need to answer a few multiple-choice questions that would take 2 minutes to complete, or do they have to upload screenshots and fill in a long survey with many free-form responses?
- The participants’ jobs
- The study location
- The duration of the engagement: longer, multi-week studies need more commitment than studies that are carried out over a few days.
Free incentive calculators (e.g., Ethnio, User Interviews, Emporia) are available online and can serve as a starting point. The final incentive number can be set after the pilot data is collected and assessed to see if the quality of responses matches the test incentive. You can ask participants during a wrapup interview to reflect on the diary study design, the incentives, and the perceived difficulty of the questions. The same questions can be added to a closing survey after the diary study, where you ask participants to rate the overall experience, fairness of the compensation, difficulty of the tasks, and give them an opportunity to leave any comments.
Tradeoff 2: Natural Behavior vs. Data Accuracy
In diary studies, we aim to capture the natural in-context setting as much as possible. But the act of logging one’s activities as they’re doing them is not natural: first, it interrupts the user from whatever they were trying to achieve, and second, it may prompt people to modify their behaviors to match the expectations that they may have about the study goals.
For example, suppose a participant is logging in-person grocery shopping. In that case, because they are busy with their diary, they might forget to buy some items. Alternatively, they may prepare in advance with an unusually detailed list that does not reflect their normal behavior because they want to do well in the study. (Diary studies are not immune to observer bias.) Address unlikely or surprising entries during the final wrap-up interview.
When should participants log their activities so that they are not overburdened or distracted? If you allow people to log the activities when it is convenient for them rather than during the activity or immediately after they complete it, they may forget details and may feel forced to make up information to fill out the diary entry fields. By waiting too long after the activity is finished, we push participants to recall too much information.
Consider the amount of information that your participants must submit with their entries. First, limit the questions to only absolutely necessary ones. If you’ve done that and it’s still likely that the diary logging will interrupt the participant, consider splitting the logging into two parts:
- A text message or a screenshot should be created at the time of the activity.
- A lengthy form containing the rest of the questions could be done at the end of the day.
Allow people to refer to the artifact they sent as they were doing the activity in order to be able to refresh their memory. If, during the pilot or the study, you notice that people tend to share their responses in bulk at the end of the week instead of making daily submissions, then set clear guidance for the entry cadence and send reminders if you notice that participants are falling behind with their submissions.
Finally, customize and adapt the format of possible participant responses, ranging from audio and video recordings to open-ended or closed-ended questions. For some participants, it might be easier and less unnatural to send quick audio notes; for others, written forms are preferred.
Tradeoff 3: Natural Behavior vs. Data Repetition
If you want to get a sense of the most frequent activities related to a particular topic, you will run a quantitative diary study with a large sample size to get specific trends. If your goal is to collect a range of moments of frustration and delight during daily device usage, you will gain valuable insights from a smaller qualitative study.
In a qualitative diary study, you usually do not need multiple entries with identical interactions from the same participant as you would not be able to quantify and generalize that data. For example, if you are conducting a smartwatch study, you do not want to see 5 out of 5 weekly entries about setting reminders (unless that was the sole purpose of the study).
Avoid an avalanche of repetitive data by limiting entries to unique interactions. To demonstrate what unique interactions mean, you will need to show participants several examples of good diary entries that include details, images, and explanations of the context with a variety of activities. For instance, with a smartwatch, people can set timers with hands-free audio controls, receive notifications, check maps, do a quick search, track health, and make payments. Setting up those limitations and boundaries in a diary-study design allows us to collect meaningful data and makes it easier for the participants to fill them out. But it also increases the chances that you are going to miss that insightful interaction that was a little different from all the others of the same type that the participant already submitted. It also sacrifices some naturalism, as participants are inherently forced to reflect on their range of activities and potentially seek out new interactions just for the sake of the diary study.
Conclusion
Before launching a diary study, acknowledge the tradeoffs that you need to make and include them in your study plan. When addressing the incentives, data accuracy, and data repetition, remember your main research question. Test your setup in a pilot study and adjust accordingly. If you did not have a chance to do a pilot, your best bet is to document your decisions and conduct a retrospective with your team to see what worked and what can be improved next time.
To recap, here’re our key recommendations:
- Set low and high incentive limits.
- Test incentive amounts.
- Address unlikely or surprising entries during the final wrapup interview.
- Avoid an avalanche of repetitive data by limiting entries to unique interactions.
References
Bolger, N., Davis, A. and Rafaeli, E. (2003) ‘Diary methods: Capturing life as it is lived,’ Annual Review of Psychology, 54(1), pp. 579–616. doi:10.1146/annurev.psych.54.101601.145030.
