Nielsen Norman Group Finds AI-Generated Designs Improve With Synthesized Research
AI-generated designs are better when based on synthesized findings and recommendations, according to Nielsen Norman Group. This matters as AI-generated prototypes become more common in design work.
Original reporting · nngroup.com
Original: “Can AI Generate Good Designs from Raw User Data?”
Read “Can AI Generate Good Designs from Raw User Data?” at Nielsen Norman Group →
How this was written. Awesome B2B Design wrote this article with a language model from Nielsen Norman Group’s reporting (model: @cf/meta/llama-3.3-70b-instruct-fp8-fast); it describes that reporting rather than adding to it. The original, linked above, is the full story.
Key points from Nielsen Norman Group’s report
- AI-generated designs are better when based on synthesized findings and recommendations, rather than raw user comments.
- Synthesized research can help AI systems generate designs that meet more scorecard criteria.
- Raw user data often lacks clear solutions and can be messy and contradictory, making it difficult for AI systems to generate good designs.
- Synthesized findings and recommendations can add a layer of interpretation that guides AI systems in making design decisions.
- The quality of AI-generated designs can depend on the quality of the input they receive.
The ability of artificial intelligence to analyze user data and generate interfaces has raised questions about its potential to replace human designers. Nielsen Norman Group reported that AI can now handle these tasks, but the quality of the output depends on the input it receives.
A recent study examined whether AI-generated designs are better when based on raw user comments or synthesized findings and recommendations. The study found that designs generated from synthesized research were superior, meeting more scorecard criteria than those based on raw user comments.
The study involved giving an AI system two design challenges, with the same prompt and design-system information, but different types of research context. The AI system generated multiple designs for each challenge, which were then scored by a human grader and a language model judge.
The results showed that designs generated from synthesized findings and recommendations met an average of 77.9% of scorecard criteria, compared to 69.5% for designs generated from raw user comments. This difference was statistically significant, according to the study.
The improvement was more pronounced in one of the design challenges, where the AI system generated designs that included important information, such as the impact of a flight cancellation on loyalty miles. In contrast, designs generated from raw comments often lacked this information.
The study's findings suggest that synthesizing research data into findings and recommendations can help AI systems generate better designs. This is because raw user data often describes problems, but does not provide clear solutions, and can be messy and contradictory.
Synthesized findings and recommendations add a layer of interpretation that can guide AI systems in making design decisions. This can help AI tools weigh user feedback against other factors, such as business needs and technical constraints.
Questions and answers
What is the benefit of using synthesized findings and recommendations in AI-generated design?
Synthesized findings and recommendations can help AI systems generate better designs by adding a layer of interpretation that guides design decisions.
Why is raw user data not sufficient for AI-generated design?
Raw user data often describes problems, but does not provide clear solutions, and can be messy and contradictory, making it difficult for AI systems to generate good designs.
What are the implications of this study for product teams?
Product teams should not assume that raw research data is sufficient context for AI tools, and should create artifacts that guide AI tools, in addition to those created for human teammates.



