arXiv:2608.09723cs.CV2026-08

让AI像人一样反复检查并修正点击位置,提升小目标和复杂界面的识别准确率。

LookAgain: Closed-Loop GUI Grounding with Visually Grounded Reflection

论文配图:LookAgain: Closed-Loop GUI Grounding with Visually Grounded Reflection
图 1 · 摘自论文原文
  • 引入闭环反思机制,每步预测后用新视觉信息检验并修正坐标。
  • 在多个基准上实现领先性能,小目标和密集控件场景提升显著。
  • 适合需要高精度交互的自动化测试与无障碍辅助工具开发。

当前的图形用户界面(GUI)定位模型在标准基准上表现优异,但在小目标、密集控件及分布外界面时性能急剧下降。我们发现其根源在于现有方法缺乏对已生成坐标的反思与修正能力:预测一旦发出即被冻结,视觉证据用于支持下一步而非检验已有结果,迭代优化聚焦于区域放大而非坐标修正。为此,本文提出LookAgain,一种基于事后视觉反思的闭环GUI定位框架。该框架将定位过程重构为多轮“预测-观察-再思考-修正”循环,包含两个核心操作:'locate' 将坐标作为假设提交,渲染标记并提取局部图像作为空间先验;'confirm' 判定假设是否成立并终止流程。通过构造反思轨迹进行监督微调(SFT),再以最终定位正确性为唯一奖励信号使用GRPO进行强化学习训练。大量实验表明,LookAgain在拒绝感知与通用GUI定位基准上均持续提升性能,达到当前最优水平。全面消融实验验证了该框架的有效性。

原文摘要 · Abstract (English)

Recent graphical user interface (GUI) grounders have significantly advanced single-shot accuracy on standard benchmarks, yet their performance degrades sharply on small targets, densely packed controls and out-of-distribution interfaces. We attribute this gap to a paradigmatic limitation shared by existing approaches: none of them treats a produced coordinate as a hypothesis to be reflected upon and revised under new visual evidence. This manifests as three coupled issues: 1) Lack of post-hoc reflection. The prediction is frozen at the moment of emission, leaving no internal mechanism to challenge or refine it. 2) Visual evidence decoupled from the prediction. The auxiliary visual evidence is gathered to support the upcoming coordinate rather than to scrutinise the one already committed to. 3) Refinement over views, not over predictions. The iterative zoom-in refines the inspected region instead of inheriting a previous coordinate as a spatial prior to be corrected. In this paper, we propose LookAgain, a closed-loop GUI grounder driven by post-prediction visual reflection. LookAgain reformulates grounding as a multi-turn predict-look-again-refine process with two primitives: "locate" posts a coordinate hypothesis, renders a marker on the image and appends a local patch of the predicted region. It anchors the next reasoning step to the previous prediction as a spatial prior; "confirm" accepts or reject the hypothesis and terminates the procedure. We train the LookAgain grounder with SFT on constructed reflective trajectories as a cold start, followed by GRPO with terminal grounding correctness as the sole reward. Extensive experiments show that LookAgain consistently improves performance on both refusal-aware and general GUI grounding benchmarks, achieving state-of-the-art results. Comprehensive ablations further verify the effectiveness of the proposed framework.

GUI定位闭环推理视觉反思智能交互

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。