Qualitative Evaluation of LLM-Designed GUI

As generative artificial intelligence advances, Large Language Models (LLMs) are being explored for automated graphical user interface (GUI) design. This study investigates the usability and adaptability of LLM-generated interfaces by analysing their ability to meet diverse user needs. The experiments included utilization of three state-of-the-art models from January 2025 (OpenAI GPT o3-mini-high, DeepSeek R1, and Anthropic Claude 3.5 Sonnet) generating mockups for three interface types: a chat system, a technical team panel, and a manager dashboard. Expert evaluations revealed that while LLMs are effective at creating structured layouts, they face challenges in meeting accessibility standards and providing interactive functionality. Further testing showed that LLMs could partially tailor interfaces for different user personas but lacked deeper contextual understanding. The results suggest that while LLMs are promising tools for early-stage UI prototyping, human intervention remains critical to ensure usability, accessibility, and user satisfaction.

💡 Research Summary

Title: Qualitative Evaluation of LLM‑Designed GUI

Purpose and Context
The paper investigates whether state‑of‑the‑art large language models (LLMs) can automatically generate usable graphical user interfaces (GUIs) from textual prompts. While generative AI has shown promise in producing visual layouts, the authors aim to assess the practical usability, accessibility, and aesthetic quality of the generated interfaces, especially for diverse user personas.

Models, Tasks, and Prompt Design
Three models released in early 2025 were evaluated: OpenAI GPT‑o3‑mini‑high, DeepSeek R1, and Anthropic Claude 3.5 Sonnet. All models received the same zero‑shot prompts that defined the model’s role as a UI expert and listed functional requirements. Three UI complexities were defined:

1. FixLine – a simple chat window with photo and location sharing.
2. FixTeam – a technical‑team panel allowing report viewing, status changes, and comments.
3. BoardPanel – a management dashboard showing system activity statistics, performance trends, support‑team KPIs, and most‑common problems, with bilingual (Polish/English) support.

Each output was a single self‑contained HTML file (HTML + CSS + JavaScript) that could be opened in a browser. The mock‑ups included basic interactivity (button clicks, dynamic content updates, language toggle) but no back‑end integration.

Evaluation Methodology
The authors combined two well‑known assessment frameworks:

• Nielsen’s 10 Heuristics – scored on a three‑point scale (meets, does not meet, indeterminate).
• Vitruvian‑triad‑derived architectural categories – six criteria (objectivity, efficiency, integrity, responsibility, regard, motivation) scored on a five‑point scale.

Additional binary checks measured whether the interface met the prompt’s element list and whether basic accessibility (sans‑serif font, sufficient contrast, readable size) was satisfied. Errors in the generated code were classified as minor, significant, or critical. Three independent experts from psychology, sociology, IT, UX, and architecture performed blind scoring, followed by consensus discussion.

Key Findings

• Layout Generation – All three models reliably produced clean, well‑structured spatial layouts. GPT‑o3‑mini‑high showed the best logical flow, likely due to its “reasoning mode” that broke the problem into sub‑steps.
• Interactivity – Functional elements (e.g., message‑send buttons, filters) were often incomplete. DeepSeek R1 generated data structures but failed to bind them correctly; language switching was inconsistent across all models.
• Accessibility – Approximately one‑third of the mock‑ups failed basic contrast or font‑size checks, indicating that LLMs do not automatically enforce WCAG guidelines.
• Aesthetics & User Needs – Claude 3.5 Sonnet produced syntactically flawless code but tended toward conservative visual choices and omitted several dashboard components, scoring lowest on the architectural criteria.
• Error Severity – Critical errors (e.g., completely broken HTML) were rare; most issues were minor (layout quirks) or significant (non‑functional interactive widgets).

Model‑Specific Observations

• GPT‑o3‑mini‑high – Strongest overall, especially on the dashboard task; excels when prompts are sequenced (role → problem → expected outcome).
• DeepSeek R1 – Fast layout generation but suffers from server overload during the experiment, leading to long latency and occasional malformed output.
• Claude 3.5 Sonnet – Consistently syntactically correct but less adaptable to external libraries; limited in meeting the full set of functional requirements.

Conclusions and Implications
LLMs are promising for early‑stage UI prototyping: they can quickly produce a coherent visual skeleton from natural‑language specifications. However, achieving production‑level usability, accessibility, and contextual relevance still requires human expertise. Prompt engineering is critical; limiting each prompt to 3‑5 clear requirements and placing high‑priority instructions at the end improves output quality. Future work should explore integrating accessibility standards into LLM training data, developing hybrid human‑AI design workflows, and extending evaluation to larger user studies.

Overall Assessment
The study provides a thorough, multi‑disciplinary evaluation of LLM‑generated GUIs, highlighting both the strengths (structured layout creation) and the current shortcomings (interactive fidelity, accessibility compliance, deep user‑context understanding). It underscores that while generative AI can accelerate the design pipeline, human designers remain indispensable for ensuring that interfaces are usable, inclusive, and aligned with real‑world user needs.