让AI在电脑上自主完成复杂任务,准确率提升至48.9%。
ComputerRL: Scaling End-to-End Online Reinforcement Learning for Computer Use Agents
- 融合API与界面操作,统一机器与桌面环境的交互方式。
- 通过分布式训练和新策略,实现超长周期稳定强化学习。
- 适合研究通用桌面自动化或构建智能办公助手的人群。
我们提出ComputerRL框架,使代理能自主操控复杂数字工作空间。该框架采用API-GUI范式,统一程序化调用与直接图形界面交互,缓解机器代理与以人为中心的桌面环境之间的不匹配问题。规模化端到端强化学习对提升多样桌面任务的性能与泛化能力至关重要,但长期训练中存在环境效率低和不稳定等挑战。为此,我们构建了可调度数千个并行虚拟桌面环境的分布式强化学习基础设施,加速大规模在线强化学习。此外,提出Entropulse训练策略,通过交替进行强化学习与监督微调,有效缓解长时间训练中的熵崩溃问题。我们在开源模型GLM-4-9B-0414和GLM-4.1V-9B-Thinking上应用ComputerRL,于OSWorld基准测试中评估,AutoGLM-OS-9B达到48.9%的新最优准确率,显著提升通用代理在桌面自动化中的表现。代码与新构建的OfficeWorld基准已开源(https://github.com/thudm/ComputerRL),该算法与框架已被用于构建AutoGLM(Liu et al., 2024b)。
原文摘要 · Abstract (English)
We introduce ComputerRL, a framework for autonomous desktop intelligence that enables agents to operate complex digital workspaces skillfully. ComputerRL features the API-GUI paradigm, which unifies programmatic API calls and direct GUI interaction to address the inherent mismatch between machine agents and human-centric desktop environments. Scaling end-to-end RL training is crucial for improvement and generalization across diverse desktop tasks; however, it remains challenging due to environmental inefficiency and instability during extended training. To support scalable and robust training, we develop a distributed RL infrastructure capable of orchestrating thousands of parallel virtual desktop environments to accelerate large-scale online RL. Furthermore, we propose Entropulse, a training strategy that alternates reinforcement learning with supervised fine-tuning, effectively mitigating entropy collapse during extended training runs. We employ ComputerRL on open models GLM-4-9B-0414 and GLM-4.1V-9B-Thinking, and evaluate them on the OSWorld benchmark. The AutoGLM-OS-9B achieves a new state-of-the-art accuracy of 48.9%, demonstrating significant improvements for general agents in desktop automation. Our code and the new OfficeWorld benchmark are available at https://github.com/thudm/ComputerRL. The algorithm and framework are adopted in building AutoGLM (Liu et al., 2024b).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。