arXiv:2606.24515cs.AIcs.HC2026-06中稿 · the 4th Internatio…

用AI自动评估用户任务完成度,让电脑操作代理更高效地学习。

Reinforcement Learning for Computer-Use Agents with Autonomous Evaluation

论文配图:Reinforcement Learning for Computer-Use Agents with Autonomous Evaluation
图 1 · 摘自论文原文
  • 用视觉语言模型自动判断任务是否完成,替代人工设计奖励函数。
  • 纠正评估器的错误反馈后,成功率平均提升12.6个百分点。
  • 适合想在无标注环境下训练桌面操作智能体的研究者和开发者。

计算机使用代理(CUAs)通过直接感知和操作图形用户界面来执行用户的高层目标。然而,由于开放桌面环境通常缺乏可扩展、机器可读的奖励信号,基于强化学习的CUA训练仍具挑战:任务成功往往依赖视觉判断,难以通过手工设计的奖励函数或密集的人工标注来定义。本文提出一种强化学习微调框架,利用自主的视觉-语言评估作为可扩展的监督信号。给定最终截图和原始指令,视觉-语言模型无需任务特定启发式或人工标签即可判断任务完成情况并提供终端反馈。由于自主评估器存在误差,我们将其反馈建模为噪声二元奖励通道,并推导出适用于近端策略优化的噪声修正奖励估计器。在macOSWorld、Windows Agent Arena和OSWorld上的实验表明,经过修正的评估器奖励优于零样本基线和原始评估器奖励,在平均成功率上分别比零样本性能高出12.6个百分点,比原始评估器微调高出5.1个百分点。结果表明,当显式建模并校正评估噪声时,自主评估可成为GUI环境中强化学习的实用奖励信号。

原文摘要 · Abstract (English)

Computer-Use Agents (CUAs) execute high-level user goals by perceiving and acting directly within graphical user interfaces. However, reinforcement learning for CUAs remains difficult because open-ended desktop environments rarely provide scalable, machine-readable reward signals: task success is often visually grounded and hard to specify with handcrafted reward functions or dense manual labels. We propose an RL fine-tuning framework that uses autonomous vision-language evaluation as a scalable supervision signal for GUI agents. Given a final screenshot and the original instruction, a Vision-Language Model judges task completion and provides terminal feedback without task-specific heuristics or manual labels during policy optimization. Because autonomous evaluators are imperfect, we model their feedback as a noisy binary reward channel and derive a noise-corrected reward estimator for Proximal Policy Optimization. Experiments across macOSWorld, Windows Agent Arena, and OSWorld show that corrected evaluator rewards outperform both zero-shot baselines and raw evaluator rewards, improving success rates by an average of 12.6 percentage points over zero-shot performance and 5.1 points over raw evaluator fine-tuning. These results suggest that autonomous evaluation can serve as a practical reward signal for RL in GUI environments when evaluator noise is explicitly modeled and corrected.

强化学习视觉语言模型自动化评估桌面代理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。