This sidebar outlines the methodology for the context-design study described in our article, Can AI Generate Good Designs from Raw User Data?
Research Question
Many product teams now use AI tools to generate interfaces. Teams hand these tools different inputs, from a one-line prompt to a full research-findings deck, and expect good outputs. Our main research question was:
Does the type of context given to an AI tool affect the quality of the interface it generates?
Within this question, we explored the following comparisons:
- Research context versus no research context. Does giving AI any research context, raw or synthesized, lead to better designs than giving it no research context?
- Raw user comments versus no research context. Does giving AI raw user comments improve designs?
- Synthesized research versus raw user comments. Do findings and recommendations produce better designs than raw user comments?
- Traditional deliverables versus Markdown files. We gave AI the same research findings packaged three ways: as a persona, a findings deck, or a Markdown file. Does the packaging change the quality of the designs?
- Findings with and without raw user comments. Does adding the raw user comments to the findings file lead to better designs?
- Explicit versus implicit knowledge. Does research context help AI tools only with the specific problems and solutions it describes, or does it also help AI apply general UX best practices?
Experimental Design
Independent Variables
The study had two independent variables (6 × 2 factorial design).
Independent Variable 1: Research Context
AI design tools can receive many kinds of context, such as design systems, product requirements, and user research. This study focused on research context. Every design received the same task prompt and design-system file. The first independent variable was the type of research context the AI received, with 6 levels ranging from no research to raw user comments and synthesized findings.
|
Level |
Description |
Input |
|---|---|---|
|
Level 0 |
No research context |
Task prompt + design.md |
|
Level 1 |
Raw user comments |
Task prompt + design.md + raw user comments |
|
Level 2a |
Persona |
Task prompt + design.md + persona PNG |
|
Level 2b |
Findings deck |
Task prompt + design.md + findings deck PDF |
|
Level 3 |
Findings Markdown file |
Task prompt + design.md + findings Markdown file |
|
Level 4 |
Findings Markdown file + raw user comments |
Task prompt + design.md + findings Markdown file + raw user comments |
Independent Variable 2: Design Task
The second independent variable was the design task: either a flight-rebooking form or an ecommerce-checkout flow. We tested two tasks to see whether the effect of research context held across different kinds of design tasks.
|
Task |
Description |
User-Comment Sources |
|---|---|---|
|
Flight rebooking |
A one-page form that lets passengers resolve a canceled flight by rebooking or requesting a refund or travel credit |
|
|
Ecommerce checkout |
A multistep checkout flow from cart to confirmation for a first-time shopper buying a gift on her phone |
We generated 8 designs for each combination of research context and task, for a total of 96 designs.
Dependent Variable
The dependent variable was design quality, measured as the percentage of applicable scorecard criteria each design met.
Procedure
Stage 1: Generating Each Design
1. We built the research context. Each task had its own set of raw user comments, and every level that included research context drew on the same dataset. To create the dataset, we used a fresh session to select user comments related to each task. For the flight-rebooking form, the AI randomly selected 100 tweets from the dataset's Canceled Flight and Flight Booking Problems categories. For the ecommerce-checkout flow, the AI tool searched app reviews for checkout-related keywords and chose 65 comments. We then added 40 unrelated reviews as noise.
A researcher analyzed the raw user comments with AI assistance in a fresh session (Claude Opus 5.5) and wrote the findings and recommendations. There were 9 findings and recommendations for the flight-rebooking form and 11 for the ecommerce-checkout flow. Each finding described a problem users experienced, the number of comments supporting it, and a design recommendation. We saved the same findings as both a findings-deck PDF (Level 2b) and a Markdown file; the Markdown file was used on its own in Level 3 and combined with the raw user comments in Level 4. The findings deck followed the traditional structure of UX-findings decks, with a quote for each finding, an agenda, a methodology slide, and next steps.
We then built personas from the findings and saved them as PNG images (Level 2a). The flight-rebooking form had 2 personas, and the ecommerce-checkout flow had 1 persona. Findings and recommendations were reworded and worked into the personas as behaviors, frustrations, goals, and key tasks on the page. Each persona also had a short bio. The personas didn’t include comment counts, while the findings deck (Level 2b) and Markdown file (Level 3) did.
2. We wrote the prompts. Every prompt had the same task and the same design-system file (design.md). The prompts differed only in the research files that they pointed to.
3. We generated the designs. Each design was made in two steps using a setup called orchestration. First, a planning model (Claude Fable 5, high-effort) read the task and the research and wrote a detailed design spec. Then a builder model built the prototype from that spec alone. The builder model never saw the research files and was instructed to build exactly what the spec described, without adding or leaving out anything. Splitting the work this way kept costs down, because Claude Fable 5 made the design decisions and a less expensive model wrote the code.
The builder model depended on the task complexity. The flight-rebooking form was a single page, so Claude Sonnet 5 could build it reliably. The ecommerce-checkout flow had several steps, so it needed Claude Opus 5.5 to avoid coding bugs.
Since AI is nondeterministic, we ran each level 8 times, and each trial produced its own design. We wrote scripts that ran the trials one after another in Claude Code’s headless mode, meaning the script gave Claude Code the prompt and Claude Code worked through the task without a person typing replies or approving steps. Each trial started in a fresh session with no access to files from earlier trials.
4. We packaged the designs for blind grading. Each design went into a packet (a folder with only what a grader needed) under a random ID. Each packet held three files:
- Prototype.html: the design
- Task_brief.txt: the task for the grader, with no information about research context
- Test_credentials.txt: credentials to test the design (e.g., a sample booking-confirmation code and last name for the flight-rebooking form, a test credit-card number for the ecommerce-checkout flow)
A private file linked each random ID to its task, level, and trial. It stayed hidden until grading was done, so graders didn’t know which research context produced each design.
Stage 2: Design-Quality Evaluation
1. We wrote the scorecards. Each task had its own scorecard — a list of criteria that a good design should meet, such as “every checkout step shows the items being bought.” To limit bias, senior managers at NN/G who weren’t involved in the study wrote them. They wrote the criteria while reviewing 5–10 sample designs for each task, without knowing which context level produced each one. The sample designs came from an early pilot generation, separate from the 96 graded designs, and included designs from various levels. The scorecard writers did not see the research findings or any other study materials while creating the criteria. The flight-rebooking scorecard had 27 scored criteria, and the ecommerce-checkout scorecard had 33.
The initial wording of some criteria was not specific enough to be graded by an LLM judge. We reworded those criteria to make the pass-or-fail decision unambiguous.
2. Two graders scored every design. A human grader on the study team opened every design in a browser and clicked through the design as a user would, while filling out the scorecard. An LLM judge (Codex GPT-5.6-Terra, high effort) read each design’s HTML code, task brief, and credentials in a fresh session for every packet and wrote evidence for each criterion it marked. The two graders worked separately, and the LLM judge never saw the human grader’s scores. Each criterion was marked “present,” “absent,” or “could not determine.” “Could not determine” was used when a criterion didn’t apply to a design, such as a criterion about marking fields as “optional” in a form that had no optional fields. That criterion was excluded from the design’s score. This came up 10 times across the 96 designs and 2,880 scoring decisions. A design's final score was the percentage of applicable scorecard criteria it met.
3. We resolved disagreements between the graders. Where the two graders disagreed, we went back to the design to decide which score was right. The human grader, who clicked through each design, caught some mistakes by the LLM judge. The LLM judge, which quoted evidence for every score, caught details the human grader overlooked.
When a disagreement came from a criterion that was vague, the researcher reworded it to make the pass-or-fail decision unambiguous, and the designs were regraded with the new wording. All rewording happened while designs were identified only by random IDs. After this process, the two graders agreed on about 96% of criteria.
4. We unblinded the results. Once grading was finished, we matched each random ID to its task and level.
Statistical Analysis
All analyses used each design’s score (the percentage of applicable scorecard criteria that the design met), with 8 designs per context level for each design task.
Comparisons Among Context Levels
We first analyzed the two design tasks together, using a two-way ANOVA with context level (6 levels) and design task (2 tasks) as factors. The interaction between the two factors tested whether the effect of context level differed between design tasks. We then compared every combination of context level and design task using Tukey post hoc tests, which adjusted for all 66 pairwise comparisons.
We also ran a one-way ANOVA across the 6 context levels separately for each design task, followed by Tukey post hoc tests that compared every pair of levels.
We used these pairwise comparisons to address four of the comparisons listed in the Research Question section: research context versus no research context, raw user comments versus no research context, traditional deliverables versus Markdown files, and findings with and without raw user comments.
Planned Contrasts: Synthesized Findings vs. Raw User Comments
To test the effect of synthesized findings compared with raw user comments, we conducted two planned contrasts:
(1) raw user comments (Level 1) versus the average of the findings deck (Level 2b) and the findings Markdown file (Level 3)
(2) raw user comments (Level 1) versus the average of the findings deck (Level 2b), the findings Markdown file (Level 3), and the findings Markdown file plus raw user comments (Level 4)
Each contrast tested the difference between the mean of the findings levels and the mean of raw user comments (Level 1), using the error term from the corresponding ANOVA. We ran the same contrasts on the two design tasks combined, using an ANOVA with context level and design task as factors, as well as separately for each design task. We report each contrast as a mean difference in percentage points with a 95% confidence interval.
We excluded personas (Level 2a) from the planned contrasts for two reasons: the personas differed in content from the findings deck and the Markdown file, and Level 2a was the only context level whose effect differed significantly between design tasks. Including them could have skewed the comparison between synthesized findings and raw user comments.
Follow-up Analysis: Explicit vs. Implicit Knowledge
After the main study, we analyzed the data further to see if research context helped only with the problems the findings and recommendations described (explicit knowledge), or also with the application of general best practices they didn’t mention (implicit knowledge). One researcher rated each scorecard criterion based on whether the findings file clearly stated what the criterion asked for (yes, partial, or no).
We then compared how much designs improved on criteria included in the findings vs. criteria they never mentioned. We compared designs generated with no context (Level 0) to designs generated with the findings (Levels 2b, 3, and 4 combined) by conducting linear mixed-effects models. The outcome was whether each criterion was met (1 = met, 0 = not met). The key term was the interaction between context level and whether the criterion was explicitly stated, which estimates how much more the stated criteria improved than the unmentioned ones. Each model included random effects for each generated design and for each criterion. We ran the comparison separately for each design task. To test whether the effect differed across design tasks, we also ran a separate model that included a three-way interaction among context level, whether the criterion was explicitly stated, and design task. We then repeated the analysis comparing Level 0 to the lighter-context levels (1 and 2a combined), which didn’t include the findings. As a robustness check, we compared each criterion’s change from Level 0 between stated and unstated criteria using Mann-Whitney tests, which led to the same conclusions for each task.
Limitations
- We tested one planning model, Claude Fable 5. This is the model that read the research and made the design decisions. Other models may use the same research differently.
- The builder model differed between the two tasks (Claude Sonnet 5 for the flight-rebooking form, Claude Opus 5.5 for the ecommerce-checkout flow). Differences between tasks may come from the builder model as well as from the task.
- The research reached each design only through the planning model’s spec; the builder model never saw the research files. Tools that generate an interface in a single step may use the same research differently.
- Models change quickly. These results reflect the state of Claude Fable 5 in October 2026.
- The design tasks were synthetic. Our team wrote each one and built the research-context materials from public datasets. We selected comments for the online checkout task rather than sampling them randomly.
- The raw datasets were small (about 100 short public comments each). We didn’t test larger datasets, such as hours of interview transcripts.
- One researcher wrote all the findings. A different researcher might have synthesized the same data differently.
- AI tools are nondeterministic. The same context can produce different outputs. We ran 8 trials per level to account for this, but small differences may still come from that variation.
- The findings deck (Level 2b) was built like a traditional findings deck, including user quotes, an agenda, methodology, and next steps. Differences between the findings deck (Level 2b) and the findings Markdown file (Level 3) may come from those extras as well as the file format.
- The information in the personas (Level 2a) differed from the information in the findings deck and Markdown file. Personas didn’t include comment counts, and reworded the findings and recommendations into goals, frustrations, and key tasks, and added an invented bio. Differences between the persona level and the other findings levels may come from those differences.
- The scorecards were written from a small set of sample designs from an early pilot, which covered a range of context levels. The criteria may favor features those sample designs happened to include.
- The human grader was the researcher on the study team. Her expectations could have influenced the scores. To limit this, she graded blind, without knowing which context level produced each design, and an independent LLM judge scored every design as a check.
- We searched the session logs from every trial for evidence that the planning model opened each research file. For text files (the CSV data and the Markdown findings file), we could confirm that it read the whole file. For the findings deck (PDF) and the persona (PNG), we could confirm only that the model opened the file.