arXiv:2607.04334cs.AI2026-07

诊断GUI智能体是相信看到的像素还是依赖结构,发现多数模型误信过时结构。

Do GUI Agents Believe Their Eyes? Diagnosing State-Belief Reliance on Pixels versus Structure

论文配图:Do GUI Agents Believe Their Eyes? Diagnosing State-Belief Reliance on Pixels versus Structure
图 1 · 摘自论文原文
  • 通过单通道干预测量模型对像素与结构的依赖程度
  • 88%的未编辑快照中模型仍跟随过时结构,任务失败率超97%
  • 训练无关的一致性门可同时降低劫持与错误,适合安全部署

多模态GUI智能体通过截图像素和序列化结构(如DOM或无障碍树)双重渠道读取界面。在行动前,智能体会形成当前界面状态的信念,但现有基准仅评估任务成功率、元素定位或抗攻击能力,未考察该信念是否源自像素。本文提出视觉状态依赖性,量化信念来自像素、结构或先验的贡献,并通过735个探针(涵盖真实网页、移动端和桌面端,其中225个来自生产环境的零修改差异)进行配对单通道干预测试,所有结果由确定性强制选择评分,无模型裁判。核心指标为感知融合差距(PFG),即模型正确感知但冲突时仍偏向结构的比例;在目标区域紧裁剪后该差距依然存在。跨四大厂商模型,文本状态信念普遍倾向结构,而纯图像准确率接近满分;在未更新的静态快照上,相同模型在高达0.88的探针中仍遵循过时结构。白盒消融显示文本影响源于单一复制的结构值,梯度归因表明视觉证据被处理但被覆盖。在多步环境中,首个信念误源导致任务失败,自恢复率不超过0.03。四种缓解策略对比显示,提示级线索在动作层面失效,证书检查虽能保障安全但引发拒绝,唯有一致性门(无需训练)同时减少劫持与任务错误。视觉状态依赖性提供了一种可量化的诊断方法,判断智能体状态信念是否真正基于视觉输入。

原文摘要 · Abstract (English)

Multimodal GUI agents read an interface through two redundant channels: the rendered pixels of a screenshot and a serialized structure such as a document object model or accessibility tree. Before acting, an agent forms a belief about the current interface state, but existing benchmarks score task success, element grounding, or attack resistance and do not ask whether that belief is drawn from the pixels. We formalize visual state reliance, the attribution of a state belief to pixels, structure, or priors, and measure it with paired single-channel interventions over 735 probes spanning real web, mobile, and desktop interfaces, of which 225 are zero-edit divergences mined from live production websites, all scored by deterministic forced choice with no model judge. Our central metric is the Perception-Fusion Gap (PFG), the fraction of probes a model perceives correctly yet resolves toward structure under conflict; a stricter variant that re-verifies perception on a tight crop of the target region leaves the gap intact. Across models from four vendors, textual state beliefs defer to structure while image-only accuracy stays near ceiling, and on unedited stale snapshots from live pages the same models follow the outdated structure on up to 0.88 of probes. A white-box ablation traces the textual effect to a single copied structural value, and gradient attribution shows the visual evidence is processed yet overridden. In live multi-step environments, one mis-sourced belief at the first step compounds into task failure with a self-recovery rate of at most 0.03. Comparing four mitigations on identical probes, prompt-level cues fail at the action level, certificate checks buy safety with refusals, and a training-free consistency gate is alone in reducing both hijack and task error. Visual state reliance thus gives a measurable diagnostic of whether agent state beliefs are visually grounded.

GUI代理状态信念视觉依赖安全诊断

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。