让电脑操作智能体在线学习,提升复杂任务成功率。
EvoCUA-1.5: Online Reinforcement Learning for Multi-turn Computer-Use Agents

- 分步优化策略+动态课程机制,应对多轮交互挑战
- 在OSWorld-Verified上达63.2%成功率,超越大模型基线
- 适合研究在线强化学习与自动化桌面任务的开发者
电脑操作智能体需通过与部分可观测、多模态桌面环境的持续交互完成长程任务。尽管模仿学习和离线轨迹优化提供强先验,但静态轨迹无法涵盖真实电脑使用中的因果反馈环:每个动作改变屏幕状态、未来动作空间和恢复选项。EvoCUA-1.5将自进化电脑操作智能体从离线经验学习扩展至在线强化学习,使策略在可执行沙箱环境中交互,并基于可验证的任务结果进行改进。该设定下的在线强化学习不仅需要直接复用单轮语言-强化学习方法。多轮交互引入上下文管理的观测、稀疏终端奖励、变长轨迹和缓慢环境反馈。EvoCUA-1.5提出分步策略优化(STEPO),在分解为步骤级样本后仍保持轨迹级优势平衡;基于策略的过滤与通过率校准,用于可验证合成任务;动态三适应课程(DTAC),结合可学习任务、困难正向回放与受控不可行任务暴露;以及全异步强化学习基础设施,包含过时控制与小批次批量处理。实验表明这些组件提升了训练稳定性与下游性能。EvoCUA-1.5在OSWorld-Verified上达到63.2%成功率,优于同等规模(32B/35B)的开源权重基线,甚至接近参数量大得多的模型。整体上,EvoCUA-1.5为多轮电脑操作智能体的在线强化学习提供了实用框架。
原文摘要 · Abstract (English)
Computer-use agents must solve long-horizon tasks through repeated interaction with partially observable, multimodal desktop environments. Although imitation learning and offline trajectory refinement provide strong priors, static traces cannot cover the causal feedback loop of real computer use: each action changes the screen state, future action space, and recovery options. EvoCUA-1.5 extends self-evolving computer-use agents from offline experience learning to online reinforcement learning, where policies interact with executable sandbox environments and improve from verifiable task outcomes. Online RL in this setting requires more than directly reusing single-turn language-RL recipes. Multi-turn interaction introduces context-managed observations, sparse terminal rewards, variable-length trajectories, and slow environment feedback. EvoCUA-1.5 addresses these challenges with Step-Level Policy Optimization (STEPO), which preserves trajectory-level advantage balance after decomposition into step-level samples; policy-aware filtering and pass-rate calibration over verifiable synthesized tasks; Dynamic Tri-Adaptive Curriculum (DTAC), which combines learnable tasks, difficult positive replay, and controlled infeasible-task exposure; and a fully asynchronous RL infrastructure with staleness control and mini-group batching. Experiments show that these components improve training stability and downstream performance. EvoCUA-1.5 achieves 63.2\% success on OSWorld-Verified, outperforming comparable 32B/35B-scale open-weight baselines and even approaching models with significantly larger parameter counts. Overall, EvoCUA-1.5 provides a practical framework for scaling online RL in multi-turn computer-use agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。