arXiv:2606.11078cs.AIcs.CL2026-06

让电脑操作智能体记得过去动作并看懂界面,提前纠错提升成功率。

A History-Aware Visually Grounded Critic for Computer Use Agents

论文配图:A History-Aware Visually Grounded Critic for Computer Use Agents
图 1 · 摘自论文原文
  • 用多模态模型记录历史操作并视觉校验每一步点击
  • 在网页/手机/桌面任务中分别提升5.8%和9.0%成功率
  • 适合需要长期规划的复杂界面操作任务

针对计算机使用智能体(CUAs)在复杂图形用户界面中因缺乏历史记忆和视觉理解导致的执行错误,本文提出HiViG框架。该框架通过在真实GUI轨迹上训练的多模态评论模型,将过往交互抽象为紧凑的历史记录,并结合当前屏幕截图对动作坐标进行视觉验证。测试时,系统集成该评论器于策略决策循环中,提供宏观动作历史以缓解短视规划问题,同时输出视觉引导的批评以在执行前拦截错误。在网页、移动和桌面多个基准上,HiViG相比最强基线平均成功率分别提升5.8%(Qwen3-VL-32B)和9.0%(Gemini-3-Flash),展现出优异跨平台泛化能力。消融实验表明,宏观历史与视觉校验均对长程任务中的测试时扩展至关重要。

原文摘要 · Abstract (English)

Various test-time interventions for Computer Use Agents (CUAs), including critic models, have been developed to improve performance through pre-execution action evaluation in complex Graphical User Interface (GUI) environments. However, existing critics suffer from two key limitations: they (1) focus primarily on short-sighted decision loops (e.g., forgetting earlier actions) and (2) lack the visual grounding needed to detect flawed actions (e.g., clicking wrong UI elements). To address these, we introduce HiViG, a History-aware Visually Grounded test-time framework, built around a multimodal critic trained on real GUI trajectories to abstract past interactions into a compact record and to evaluate actions with visual grounding. At test time, HiViG integrates the critic into the policy decision loop to provide macro-action history, which summarizes the policy's completed achievements, and visually grounded critique, which verifies raw execution coordinates against the current screenshot to intercept errors before execution. Across web, mobile, and desktop benchmarks, HiViG consistently outperforms existing scalar and verbal critics, improving average success rates over the strongest baseline by 5.8% for Qwen3-VL-32B and 9.0% for Gemini-3-Flash, and demonstrates strong cross-platform generalization. Ablations show that macro-action history mitigates short-sighted planning and visually grounded critique reduces execution errors, with both components being critical for test-time scaling in long-horizon GUI tasks.

GUI智能体视觉校验历史记忆

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。