GUI智能体易出错且不可逆,这篇综述系统梳理了可信性挑战与防御方案。
Towards Trustworthy GUI Agents: A Survey
- 提出感知-推理-交互三阶段信任分解框架,揭示错误传播机制
- 发现任务完成率无法反映真实可信度,需评估错误级联与安全权衡
- 总结对抗攻击与防御方法,推动建立更可靠的评估基准
图形用户界面(GUI)智能体将大语言模型从文本生成拓展至真实数字环境中的动作执行。与对话系统不同,GUI智能体执行提交表单、授予权限或删除数据等不可逆操作,可信性成为核心要求。本文识别出执行差距是构建可信GUI智能体的关键挑战:在动态、部分可观测的界面中,感知、推理与交互之间存在错位。我们提出一个流程对齐的分类体系,将信任分解为感知信任、推理信任与交互信任,揭示失败如何沿智能体流水线传播并经动作/观察循环叠加。系统回顾各阶段的良性故障模式与对抗攻击,并提出针对GUI场景的相应防御机制。进一步分析评估实践,指出仅以任务完成率为指标不足以衡量可信度。强调新兴的可信度度量与基准,可捕捉错误级联及安全与效用间的权衡,最后列出部署GUI智能体所面临的安全可靠性的开放挑战。
原文摘要 · Abstract (English)
Graphical User Interface (GUI) agents extend large language models from text generation to action execution in real-world digital environments. Unlike conversational systems, GUI agents perform irreversible operations such as submitting forms, granting permissions, or deleting data, making trustworthiness a core requirement. This survey identifies the execution gap as a key challenge in building trustworthy GUI agents: the misalignment between perception, reasoning, and interaction in dynamic, partially observable interfaces. We introduce a workflow-aligned taxonomy that decomposes trust into Perception Trust, Reasoning Trust, and Interaction Trust, showing how failures propagate across agent pipelines and compound through action/observation loops. We systematically review benign failure modes and adversarial attacks at each stage, together with corresponding defense mechanisms tailored to GUI settings. We further analyze evaluation practices and argue that task completion alone is insufficient for trust assessment. We highlight emerging trust-aware metrics and benchmarks that capture error cascades and the security/utility trade-off, and outline open challenges for deploying GUI agents safely and reliably.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。