用视觉对比提升GUI智能体的步骤验证能力，减少长流程任务失败。
VisCritic: Visual State Comparison as Process Reward for GUI Agents
- 通过对比操作前后的截图，直接判断界面变化是否正确。
- 在五个基准上提升任务成功率，错误类型识别准确率超85%。
- 无需人工标注，可无缝集成到各类GUI智能体中使用。
由视觉语言模型驱动的GUI智能体在自动化数字任务方面潜力巨大，但在长周期场景中常因缺乏步骤级验证而失败。现有过程奖励模型仅依赖文本推理，忽略了GUI状态变化的视觉特性。本文提出VisCritic，一种基于视觉特征空间的流程奖励框架，通过直接比较操作前后的屏幕截图来验证智能体动作。该框架采用孪生视觉变压器提取变化感知表征，并结合动作感知评论头，联合评估动作成功、任务进展和错误类型。通过一个弱监督数据构建管道，从现有轨迹中生成训练样本，无需额外人工标注。在五个基准上的实验与离线分析表明，VisCritic可作为即插即用的增强模块，普遍提升各类GUI智能体的性能指标，并提供可视化诊断线索。
原文摘要 · Abstract (English)
GUI agents powered by vision-language models show strong potential for automating digital tasks, yet frequently fail in long-horizon scenarios due to the absence of step-level verification. Existing process reward models verify actions through textual reasoning alone, missing the visual nature of GUI state changes. We introduce VisCritic, a visual process reward framework that verifies agent actions by directly comparing pre-action and post-action screenshots in visual feature space. VisCritic employs a Siamese vision transformer to extract change-aware representations, coupled with an Action-Aware Critic Head that jointly evaluates action success, task progress, and error type. A critic-training data construction pipeline generates weakly supervised samples from existing trajectories without additional human labels for critic training. Experiments and offline analyses across five benchmarks demonstrate that VisCritic serves as a plug-and-play enhancement for diverse GUI agents, generally improving benchmark metrics while providing visual diagnostic cues.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。