Synthetic Users: If, When, and How to Use AI-Generated “Research”

Of the new AI-based tools flooding the landscape of user and product research, few have caused as much debate as AI-generated users — also known as “synthetic users”.

TL;DR

• Real user research is essential. Synthetic users cannot replace the depth and empathy gained from studying and speaking with real people. They often provide shallow or overly favorable feedback.
• Supplement, don’t substitute. If you’re using synthetic users in your research process, they should complement, not replace, real research.
• Use synthetic research for specific purposes. Synthetic users are useful for desk research and generating hypotheses, but not for final decision-making.

What Are Synthetic Users?

A synthetic user is an AI-generated profile that attempts to mimic a user group, providing artificial research findings produced without studying real users. A synthetic user will express simulated thoughts, needs, and experiences.

A synthetic user is based on large language models (LLMs) that have been trained on vast amounts of data about people.

While general-purpose AI chatbots like ChatGPT can be used to create synthetic users, there are also dedicated AI-based platforms designed for this task — perhaps the most famous being the company named Synthetic Users. (In this article, we use capitalization to refer to this company’s product and lowercase to refer to the concept of synthetic users.)

How Synthetic Users’ Product Works

In a conversation with Hugo Alves, cofounder of Synthetic Users, we learned about the product's practical applications. According to Alves, Synthetic Users is particularly useful in scenarios where swift decision making is crucial and absolute certainty isn't required. The tool allows teams to access the thoughts, reflections, and emotions of specific user groups, providing timely and frequent access to user insights.

At first glance, a Synthetic Users’ synthetic user profile looks similar to a user persona—it contains demographic details like a name, age, and education level. Like a persona, such a profile is intended to be an imaginary representation of a real group of people. However, unlike a persona, the profile is “alive” — meaning that you can chat with it.

To generate these AI-generated users, you specify the user group and the goal of your study.

Example User Group:

Medical-detailing representatives who work in Latin America

Example Goal:

How do medical-detailing representatives work?

The product then uses some dynamic parameters to create different users and uses the study goals to craft interview questions. Within a few seconds, your synthetic users have been created, and interview transcripts are ready to read. Synthetic Users provides the ability to ask followup questions and continue the interview (which we tried).

Cost-Saving Solution or the Dawn of a Research Apocalypse?

We have long argued that UX without user research isn’t UX. But, we did not anticipate that people would attempt to conduct user research without real users. We need to rephrase our slogan:

UX without real-user research isn’t UX.

In our interview, Alves stressed that he does not intend for his product to replace user research completely.

“You're never gonna stop talking to real people and you shouldn't. You shouldn't. There are some decisions that you shouldn't even go to synthetic users. You should just go to humans.” Hugo Alves, cofounder of Synthetic Users

However, the current marketing copy on the Synthetic Users site does not clearly reflect that stance.

If you know NN/g, you know that we never recommend that teams skip user research. Research with real people provides many intrinsic benefits; for example, it creates empathy and builds a vivid representation of the user in each team member’s mind (something reading a transcript from a synthetic user does not adequately do).

Putting those benefits aside, we tested synthetic users to see if there was a qualitative difference in the insights gained from real users compared to synthetic ones.

About Our Evaluation

We used Synthetic Users and ChatGPT to generate synthetic users and insights for three studies we had performed with real users. Currently, these tools are limited to replicating text-based attitudinal methods, specifically interviews and surveys. In our evaluation, we focused on interviews.

In Synthetic Users, we specified the user group and the goals for our research. The tool then generated fake people and performed simulated interviews for us.

Since ChatGPT doesn’t generate the same outputs as Synthetic Users without a lot of prompting, we began each synthetic user interview with a prompt that asked the chatbot to pretend to be a particular person.

Example prompt used with ChatGPT:

Pretend you are a medical detailing representative working in Bogotá, Colombia. For some context: You've been working in the role for 6 years. Your day-to-day work involves meeting with doctors in hospitals and clinics around Bogota. You use a CRM solution that helps you track and plan your doctor visits. Pretend you are a participant in a user research session. I will be the interviewer.

While ChatGPT and Synthetic Users produced similar content, speaking with a synthetic user generated by Synthetic Users felt a bit more like speaking to a person (although the responses were still very unhuman and wordy).

Our experimentation revealed some interesting possible use cases for synthetic users. However, we also found the results to be much less useful than many UX and product professionals claim.

Valid Use Case for Synthetic Research: Desk Research

When used responsibly by mature research teams, AI-generated user profiles can complement real user research.

One of the advantages of a synthetic user is that it can synthesize vast amounts of available data about a specific user group and present it in a digestible way.

For example, there are products and systems designed to support the work of medical-detailing representatives, academic literature regarding medical detailing, forums dedicated to medical-detailing representatives, and more. LLMs have been trained on all this data. When we asked a synthetic medical-detailing representative to describe a typical day, the result was very similar to what real users told us in our study.

Response from a synthetic user:

Sure, I'd be happy to. My day usually starts around 7 AM when I go over my schedule and prepare for the meetings I have lined up with doctors and healthcare professionals. First, I make sure I have all the necessary materials —brochures, samples of cardiovascular drugs and antibiotics, and any follow-up information from previous visits. I aim to be very organized and prepared because it helps in establishing trust and reliability with the doctors I meet.

By 8 AM, I'm usually on the road, visiting clinics and hospitals across Bogotá. I typically have about five to six appointments daily, where I discuss the latest research and benefits of the products I'm detailing. A crucial part of my job is building strong relationships, so I always make it a point to understand the specific needs of each healthcare professional and tailor my conversations accordingly. This might involve bringing personalized gifts or simply remembering details about their practice that show I’m paying attention.

(Generated by Synthetic Users)

Synthetic users could support researchers and designers working on projects in completely new domains. They could be a useful starting place for identifying  points of discussion that might  crop up in real-user-research sessions.

A synthetic user can be useful if researchers treat the output as a hypothesis to guide future research.

Limitations of Synthetic Users

There are many limitations to AI-generated research that pose extreme risks to teams relying on AI to fill in for users. In fact, there are too many for this article.

In the sections below, we discuss the limitations that we think will continue to exist regardless of near-term enhancements to these models and products.

AI Provides an Unrealistic View of Human Behavior

AI chatbots have a tendency to want to please (known as sycophancy) and they do not always model human behavior well. For example, Jeff Sauro and his colleagues found that ChatGPT was not an acceptable replacement for real users in tree testing, because it vastly outperformed most real people.

In our recent study of online training, a common theme was that learners (despite their good intentions) often start online training but don’t complete it. When we asked our synthetic users whether they completed all the courses they had taken, they responded (overoptimistically) that they had.

This tendency for idealization was a theme we observed over and over.

For example, most participants in our self-paced-learning study reported not using  discussion forums for their courses because they found the interactions contrived and not useful.

However, our synthetic users viewed them favorably and argued that this feature would be important to them in an online learning environment.

Real interviewer: Have you contributed to any forums (when they were available) when you were taking a course?

Synthetic user response: Yes, I have actively participated in forums and discussion boards when taking online courses. I've found that contributing to these forums is not only a way to deepen my understanding of the course material but also an opportunity to connect with fellow learners from diverse backgrounds and perspectives.

[Four more paragraphs about how the synthetic user asked questions and shared their professional experience and resources, and how they felt a sense of community via the forums]

(Generated by Synthetic Users)

The large discrepancy between what real and synthetic users told us in these two examples is due to two factors:

• Human behavior is complex and context-dependent. Synthetic users miss this complexity. The synthetic users generated across multiple studies seem one-dimensional. They feel like a flat approximation of the experiences of tens of thousands of people, because they are.
• Responses are based on training data that you can’t control. Even though there may be proof that something is good for you, it doesn’t mean that you’ll use it. In the discussion-forum example, there’s a lot of academic literature on the benefits of discussion forums on online learning and it is possible that the AI has based its response on it. However, that does not make it an accurate representation of real humans who use those products.

UX research explores human behavior. We need to hear about real human experiences to understand why people make specific choices, what specific contextual factors influence behavior, what’s important, and when. This cannot be learned through currently available synthetic users.

Values, Desires, and Needs Are Too Shallow

One reason we do user research is to identify which problems need to be solved and their priority. Synthetic users readily generate a long list of needs or pain points, with only a limited understanding of their priority.

For example, when a synthetic user was asked what makes an online course engaging, they responded with 7 factors. (For brevity’s sake, we include the full text for the first factor only.)

1. Interactive Content: Courses that include interactive elements such as quizzes, exercises, and practical projects engage me more deeply than those that are purely lecture-based. Interaction not only helps in reinforcing the learning but also makes the experience more dynamic and enjoyable.
2. Real-World Applications
3. High-Quality Materials
4. Community and Instructor Interaction
5. Personalized Learning Paths
6. Feedback and Progress Tracking
7. Flexibility

(Generated by Synthetic Users)

Real people care about some things more than others. Synthetic users seem to care about everything. This is not helpful for feature prioritization or persona creation.

In addition, the factors are too shallow to be useful. For example, the synthetic user lists quizzes, exercises, and practical projects as examples of interactive content, but which is better? When? Why? How can they be done well or poorly? All these questions remain unanswered, and further probing leads to further vague responses. We are left knowing that these are common needs of all online learners but have no idea how to address them or what each factor looks like. To know this, we’d have to study real users, watch them interact with real courses, and hear real stories from them.

Imagined Experiences Are Not Reliable

Synthetic research cannot produce behavioral data. AI can’t actually use a product like a human does (at least for now), so it certainly cannot have specific experiences.

As a result, fake users’ profiles tend to be more accurate in their opinions than their stories about their “past experiences.”

Some UX and product professionals are turning to synthetic users to validate or product concepts or solution ideas. Synthetic Users offers the ability to run a concept test: you describe a potential solution and have your synthetic users respond to it.

This is incredibly risky. (Validating concepts in this way is risky even with human participants, but even worse with AI.) Since AI loves to please, every idea is often seen as a good one.

Interviewer: Our team is also thinking about creating another application that could allow reps to deliver samples by drone. Would this be something you would use?

Synthetic user: [after pointing out pros and cons] As a medical detailing rep, I would find this application very useful, especially for delivering samples to doctors quickly and efficiently. The time saved from not having to deliver samples personally would allow me to focus more on building relationships and providing detailed product information to doctors. Additionally, the ability to fulfill urgent sample requests promptly would enhance the service quality we provide to healthcare professionals.

If implemented well, this drone delivery system could be a game-changer, improving efficiency, reducing costs, and providing a modern, high-tech solution to sample delivery.

(Generated by ChatGPT)

We found our synthetic users' feedback was often favorable and vague. Recognizing this issue, Alves and his team have invested substantial time and energy in reducing the AI's tendency to agree with its interlocutors or provide generic responses. Still, sycophancy remains an ongoing challenge.

Product teams should avoid using AI as a litmus test for ideas and should instead seek feedback from real users to have confidence that solutions warrant investment. That being said, a synthetic user could point out helpful considerations to bear in mind when exploring a solution.

How to Use Fake Findings Responsibly

If your team plans to use synthetic research, here’s how we recommend doing it safely, to maximize its benefits and avoid its risks.

• Use synthetic users to help you prepare for research studies with real users. Synthetic users could be useful when learning about a new user group, exploring which topics might be relevant to explore in an exploratory study, or piloting an interview guide.
• Treat the data you acquire from synthetic users as hypotheses that need testing. Synthetic user data can be useful for developing proto personas or proto journey maps (also known as hypothesis maps) that are revisited and refined after doing research with real users.
• Do not use synthetic-user research as a replacement for real-user research. AI-generated research findings can be accurate, but they are not always reliable. They cannot account for the complexity and nuance of real users’ opinions and behaviors.
• Do not present synthetic-user research findings as real-user research findings. This is an unethical way to share what you’ve learned. It creates the risk of permanently damaging the perception of user research in your organization. Always be clear about where your findings came from.
• Avoid using synthetic users if your user population is niche or specialized. For example, if you wanted to learn about a rice farmer in Vietnam, the synthetic user data may be patchy and inaccurate. Remember that LLMs are fed on the internet’s knowledge. As a result, broad consumer profiles are more likely to be accurate. Similarly, if you’re researching a well-known or well-documented topic, the results are probably more reliable.
• Understand that people’s online thinking and behaviors are likely different from real life. As a result, the data generated with synthetic users may be skewed and not reflect the population at large. For example, consider how people review things online. People aren’t very likely to go out of their way to write a Google review for something like a gas station, unless they have a very negative experience that they want to report. As a result, gas-station online ratings are often very low, but this does not represent the vast majority of experiences customers have there.
• Avoid adopting this tool if your stakeholders will see it as a replacement for user research. If you work in a low UX-maturity organization where it’s challenging to do research, synthetic users might seem like an ideal solution. Unfortunately, those environments may be the most likely to misunderstand and misuse synthetic research, and possibly become dependent on it.

Is Fake Research Better than No Research?

This question has been the subject of much debate within NN/g and also in the broader community. If a team does not have the ability to conduct real research (due to resources, skills, or inability to access its user population), are synthetic users better than nothing?

In terms of learning something about your users, synthetic research may be better than no research at all. The problem is that some of what you learn will be inaccurate or misrepresentative, and without real research, you won’t catch and correct those inaccuracies.

Making decisions without real-user research is dangerous. You run the risk that you’ll make an incorrect design or business decision and move forward, investing time and money, only to find out (too late) that it was a mistake.

We’re also extremely concerned about the likelihood that teams that start with synthetic research will get too comfortable and won’t move on to real-user research. If stakeholders feel that they’re getting some insight for a tiny investment, justifying the need for investing in real research may become difficult.  And, if and when those fake findings prove to be incorrect, they could permanently damage the company’s understanding of the value of user-centered research and design.

The synthetic-user debate feels similar to another contentious argument around AI-generated fake friends and romantic partners. On one hand, these simulated human interactions serve an important purpose — they can alleviate loneliness. While alleviating loneliness can be extremely beneficial, does it damage someone’s long-term ability to build relationships with humans? Over time, will they run the risk of becoming dependent on their always accommodating fake friend, and incapable (or unwilling) to forge a real relationship (with all the messy human inconveniences that it entails)?

Conclusion

While using synthetic users seems alluring, relying on them too much can be harmful to your organization and product. UX professionals and product teams should be wary of insights produced by synthetic users and treat them as hypotheses that need testing. Synthetic users can be used to help with initial desk research about new user groups and to generate hypotheses and ideas that should be tested through research with real people.

References

Jeff Sauro, Will Schiavone, and Jim Lewis. 2024. Using ChatGPT in Tree Testing: Experimental Results. MeasuringU. Retrieved from https://measuringu.com/chatgpt4-tree-test/.

Mrinank Sharma et al., 2023. Towards Understanding Sycophancy in Language Models. arXiv:2310.13548. Retrieved from https://arxiv.org/abs/2310.13548