构建可验证训练环境,让计算机使用智能体高效学习真实软件操作。
CUA-Gym: Scaling Verifiable Training Environments and Tasks for Computer-Use Agents

- 自动生成任务、环境和奖励函数,实现大规模可验证训练数据合成。
- 生成32,112个验证过的训练样本,在真实测试中达72.6%准确率。
- 适合研究智能体具身智能、自动化软件操作与强化学习泛化能力的学者。
基于可验证奖励的强化学习(RLVR)在数学、工具使用和软件工程等领域取得突破,但其在计算机使用智能体(CUAs)上的应用受限于高质量、可扩展训练数据的缺乏。现有手写基准覆盖范围小,而依赖大模型评分的数据集缺乏可靠验证。本文提出CUA-Gym,一个协同生成任务指令、环境状态和奖励函数的可扩展流水线:生成器构建初始与目标环境状态,判别器根据任务描述编写奖励函数,协调器驱动二者通过多轮执行迭代优化。生成的三元组经由大模型多数投票与智能体回放联合筛选,确保质量超越单任务对抗循环。为解决环境稀缺问题,进一步构建了基于真实软件使用分布的高保真模拟网页套件CUA-Gym-Hub,使训练数据规模显著提升。基于该流程,我们构建了包含32,112个验证过训练样本的数据库,覆盖110个环境。在该数据上使用GSPO训练的CUA-Gym-A3B和CUA-Gym-A17B模型在OSWorld-Verified测试中分别达到62.1%和72.6%准确率,优于同等规模的开源模型,且性能随数据量和环境多样性平滑增长。相同检查点在保留的WebArena基准上也表现更优,表明具备跨环境迁移能力。我们将开源完整合成流水线、数据集、环境和模型。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards (RLVR) has driven breakthroughs in domains such as math, tool-use, and software engineering, yet its extension to computer-use agents (CUAs) has been bottlenecked by the scarcity of scalable training data with deterministic rewards. Constructing such data for CUAs requires consistent task instruction, executable environment, and verifiable reward. However, hand-curated benchmarks achieve high reward fidelity but cover few applications and LLM-as-judge-based datasets scale broadly but lack reliable verification. We present CUA-Gym, a scalable pipeline that co-generates task instructions, environment states, and reward functions. Concretely, a Generator agent constructs the initial and golden environment states, and a separate Discriminator agent writes the reward function from the task specification. An orchestrator agent drives the two through iterative rounds upon execution. Generated tuples then pass a final filter combining LLM majority voting and agent rollouts, ensuring quality beyond the per-task adversarial loop. To address the scarcity of training environments, we further synthesize CUA-Gym-Hub, a broad suite of high-fidelity mock web applications grounded in real-world software-use distributions, expanding the scale of CUA RLVR data by magnitude. Using this pipeline, we construct CUA-Gym, a dataset of 32,112 verified RLVR training tuples grounded in 110 environments. Trained with GSPO on CUA-Gym, our CUA-Gym-A3B and CUA-Gym-A17B achieve 62.1% and 72.6% on OSWorld-Verified, outperforming prior open-source CUAs at comparable scales, with performance scaling smoothly in both data volume and environment diversity. The same checkpoints also improve on the held-out WebArena benchmark, indicating transfer beyond the training environments. We will open-source the full synthesis pipeline, dataset, CUA-Gym-Hub environments, and models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。