arXiv:2509.23866cs.LGcs.AI2025-09被引 18

通过解耦训练与自适应数据筛选,提升GUI智能体强化学习效率。

Efficient Multi-turn RL for GUI Agents via Decoupled Training and Adaptive Data Curation

  • 分模块异步设计,实现无阻塞通信与高效并行训练。
  • 在OSWorld上达42.13%成功率,较基线提升14.61%。
  • 适合关注GUI自动化与强化学习系统优化的研究者。

基于视觉-语言模型的GUI智能体在自动化桌面和移动任务方面展现出潜力,但面临两大挑战:(1) 多轮交互中与GUI环境的慢速滚动(rollout);(2) 缺乏高质量的智能体-环境交互数据用于策略学习。为此,我们提出DART框架——一种解耦的智能体强化学习训练方法,将系统拆分为四个异步模块:环境集群、滚动服务、数据管理器和训练器。该设计支持非阻塞通信、异步训练、按轮次采样轨迹及按工作节点同步模型,显著提升系统效率:滚动阶段GPU利用率提升1.6倍,训练吞吐量提升1.9倍,环境利用率提升5.5倍。为有效利用大量样本,引入自适应数据筛选机制:(1) 预收集困难任务的成功轨迹以补充在线采样稀疏的成功案例;(2) 根据任务难度动态调整滚动数量和轨迹长度;(3) 仅在高熵步骤(关键决策点)进行训练;(4) 通过截断重要性采样缓解策略滚动与更新间的分布偏移。在OSWorld基准测试中,DART-GUI-7B达到42.13%的任务成功率,相较基线模型绝对提升14.61%,优于开源最先进水平7.34%。我们将完整开源训练框架、数据及模型检查点,发布于computer-use-agents.github.io/dart-gui,期待推动开放社区中智能体强化学习训练的发展。

原文摘要 · Abstract (English)

Vision-language model (VLM) based GUI agents show promise for automating complex desktop and mobile tasks, but face significant challenges in applying reinforcement learning (RL): (1) slow multi-turn interactions with GUI environments for policy rollout, and (2) insufficient high-quality agent-environment interactions for policy learning. To address these challenges, we propose DART, a Decoupled Agentic RL Training framework for GUI agents, which coordinates heterogeneous modules in a highly decoupled manner. DART separates the training system into four asynchronous modules: environment cluster, rollout service, data manager, and trainer. This design enables non-blocking communication, asynchronous training, rollout-wise trajectory sampling, and per-worker model synchronization, significantly improving the system efficiency: 1.6*GPU utilization for rollout, 1.9* training throughput, and 5.5* environment utilization. To facilitate effective learning from abundant samples, we introduce an adaptive data curation scheme: (1) pre-collecting successful trajectories for challenging tasks to supplement sparse success in online sampling; (2) dynamically adjusting rollout numbers and trajectory lengths based on task difficulty; (3) training selectively on high-entropy steps to prioritize critical decisions; (4) stabilizing learning via truncated importance sampling for policy mismatch between policy rollout and updating. On the OSWorld benchmark, DART-GUI-7B achieves a 42.13% task success rate, a 14.61% absolute gain over the base model, and 7.34% higher than open-source SOTA. We will fully open-source our training framework, data, and model checkpoints via computer-use-agents.github.io/dart-gui, which we believe is a timely contribution to the open-source community of agentic RL training.

GUI智能体强化学习训练效率数据筛选

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。