让GUI智能体学会在错误指令下及时停止,避免盲目执行。
Do GUI Agents Know When Not to Act? Enabling Conflict-Aware Termination for Multimodal GUI Agents

- 引入冲突感知终止机制,评估指令与界面的逻辑一致性。
- 实验显示多数代理在冲突任务中仍强行执行,成功率不足40%。
- 轻量级推理阶段干预,提升识别异常场景能力,适合工业级应用。
图形用户界面(GUI)智能体被广泛用于根据自然语言指令操作界面,但真实用户可能因无心之失发出不可行指令。可靠的智能体不仅应知如何行动,更需懂得何时不应行动。本文提出CONFLICTGUI基准,涵盖指令内矛盾与指令-界面上下文矛盾,用于研究冲突感知终止。评估发现,现有代理普遍存在执行偏倚的过度服从:在可行任务上表现良好,但在冲突指令下仍盲目执行。为此,我们提出CONFLICTGUARD,一种推理阶段框架,将可行性认知与行为生成对齐。该框架包含两个耦合组件:可行性验证协议,引导智能体在行动前评估指令逻辑与界面证据;以及条件动作调制机制,促使智能体从过度服从转向终止行为。在五个主流代理上的实验表明,CONFLICTGUARD显著提升平均冲突任务成功率,同时保持常规任务性能。结果验证,轻量级推理阶段干预可显著增强GUI智能体识别不当执行场景并停止无效动作的能力。
原文摘要 · Abstract (English)
Graphical user interface (GUI) agents are increasingly used to execute natural-language instructions on user interfaces, yet real users may issue infeasible instructions due to benign mistakes. A reliable agent should not only know how to act, but also when not to act. In this work, we introduce CONFLICTGUI, a benchmark covering instruction-internal conflicts and instruction-GUI context conflicts to study conflict-aware termination. Our evaluation reveals severe execution-biased overcompliance: agents that perform well on feasible tasks often continue to execute blindly under conflicting instructions. To mitigate this behavior, we propose CONFLICTGUARD, an inference-time framework that aligns an agent's feasibility awareness with its action generation. CONFLICTGUARD contains two coupled components: a feasibility verification protocol that guides the agent to assess instruction logic and GUI-side evidence before acting, and a conditional action modulation mechanism that steers agents from over-compliant execution into termination-oriented behavior. Experiments across five widely-used agents demonstrate that CONFLICTGUARD improves average conflict task success rate significantly, while preserving normal GUI-task performance. These results validate that a lightweight inference-time intervention can substantially boost GUI Agent's competence to identify inappropriate execution scenarios and refrain from unnecessary actions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。