arXiv:2607.23263cs.AI2026-07

用四类角色协作判断任务完成度,让强化学习更准更省

SeekJudge: A Practical Reward Framework for Reinforcement Learning in Computer-Use Agents

  • 四类专用智能体通过循环协作判断任务是否完成
  • 在未见测试目标上成功率媲美甚至超过传统规则评估
  • 支持细粒度步骤判断,成本低且能处理超长轨迹

判断轨迹是否完成指令,直接影响计算机使用代理在长周期图形界面任务上的评估与强化学习训练。传统基于规则的评估难以契合人类意图,且在应用更新或在线内容漂移时失效。现有基于模型的评判器虽有改进,但仍与规则评估存在性能差距。我们提出 extbf{SeekJudge} 框架,由四类角色专用智能体——压缩、基准、搜寻与分析——通过在轨迹上执行“搜寻-分析”循环达成判定。一个种子校准的蒸馏流程训练出一个共享的 9B 大模型作为所有四类智能体的基础。在保留的强化学习测试目标上,以下游成功率为衡量标准,SeekJudge 是首个在实际中达到或超越原生规则监督效果的基于模型的奖励机制。除精度外,该框架还提供步骤级判断,运行成本远低于闭源大模型,且每次调用上下文极小,可扩展至更长轨迹。我们进一步提出了奖励服务器的通用架构改进,显著加速了强化学习中的判断过程。这些共同使基于模型的奖励成为计算机使用代理强化学习中规则监督的实用替代方案。

原文摘要 · Abstract (English)

Deciding whether a trajectory actually fulfills its instruction governs how we measure computer-use agents on long-horizon graphical-user-interface tasks and how we train them with reinforcement learning. This judgment has long relied on rule-based evaluation, which struggles to align with human intention and goes stale when an app updates or its online content drifts. Existing model-based judges attempt to address these problems but still leave a performance gap to the rule-based evaluation. We propose the \textbf{SeekJudge} framework, in which four role-specialized agents, a Condense, a Ground, a Seek and an Analyze agent, reach a verdict through a Seek--Analyze loop over the trajectory. A seed-calibrated distillation pipeline trains one specialized $9$B model to serve as the shared backbone for all four agents. Measured by downstream success rate on held-out RL test goals, SeekJudge is the first practical model-based reward to match or surpass native rule-based supervision in online RL. Beyond accuracy, SeekJudge provides step-level judgments, runs far cheaper than a closed-source large model, and keeps a small per-call context that scales to much longer trajectories. We further contribute a general architectural improvement to the reward server that speeds up judging in RL. Together these make model-based reward a practical drop-in for rule-based supervision in CUA reinforcement learning.

强化学习任务评估智能体协作奖励设计

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。