Towards Trustworthy GUI Agents: A Survey

Graphical User Interface (GUI) agents extend large language models from text generation to action execution in real-world digital environments. Unlike conversational systems, GUI agents perform irreversible operations such as submitting forms, granting permissions, or deleting data, making trustworthiness a core requirement. This survey identifies the execution gap as a key challenge in building trustworthy GUI agents: the misalignment between perception, reasoning, and interaction in dynamic, partially observable interfaces. We introduce a workflow-aligned taxonomy that decomposes trust into Perception Trust, Reasoning Trust, and Interaction Trust, showing how failures propagate across agent pipelines and compound through action/observation loops. We systematically review benign failure modes and adversarial attacks at each stage, together with corresponding defense mechanisms tailored to GUI settings. We further analyze evaluation practices and argue that task completion alone is insufficient for trust assessment. We highlight emerging trust-aware metrics and benchmarks that capture error cascades and the security/utility trade-off, and outline open challenges for deploying GUI agents safely and reliably.

💡 Research Summary

The paper surveys the emerging field of graphical user interface (GUI) agents, which extend large language models (LLMs) from pure text generation to concrete actions in real‑world digital environments. Unlike chatbots, GUI agents can perform irreversible operations such as clicking buttons, submitting forms, granting permissions, or deleting files, so a single mistake can cause tangible harm. The authors identify a fundamental “execution gap” that separates three stages of an agent’s operation: perception, reasoning, and interaction. Misalignment at any stage propagates downstream, creating error cascades that amplify risk.
To structure the analysis, the authors introduce a workflow‑aligned taxonomy that decomposes trust into Perception Trust, Reasoning Trust, and Interaction Trust. For each dimension they enumerate benign failure modes (e.g., hallucinations, plan‑execution mismatches) and adversarial attacks (visual perturbations, DOM injection, prompt injection, timing attacks). Corresponding defenses are grouped into input filtering, cross‑modal verification, uncertainty estimation, self‑checking, world‑model simulation, hierarchical planning, and action verification protocols. The survey emphasizes that defenses must be stage‑specific because upstream attacks can cascade into severe downstream consequences.
The paper also critiques current evaluation practices that rely solely on task‑completion rates. Such metrics ignore the irreversible nature of GUI actions and the security‑utility trade‑off that practitioners must balance. The authors propose new trust‑aware metrics—error‑cascade scores, uncertainty‑weighted success, and security‑utility curves—and introduce a benchmark suite (the “Trust‑Aware GUI Suite”) that includes dynamic UI changes, adversarial perturbations, and long‑horizon tasks.
Key insights include: (1) standard LLM safety mechanisms (output filtering, refusal training) assume reversible text output and do not transfer to agentic execution; (2) perception errors (visual or structural hallucinations) are tightly coupled with downstream safety violations; (3) reasoning modules lacking world models are prone to plan failure when unexpected pop‑ups appear; (4) interaction modules must handle coordinate mapping, timing, and device heterogeneity to avoid irreversible mis‑clicks.
Finally, the survey outlines open research challenges: real‑time uncertainty estimation, continuous world‑model updates for dynamic interfaces, lightweight multimodal defenses, legal accountability for irreversible actions, and user‑centric transparency mechanisms. By mapping the threat landscape, detailing defense mechanisms, and advocating richer evaluation protocols, the paper provides a comprehensive roadmap for building trustworthy GUI agents that can be safely deployed in real‑world applications.