用离线数据模拟在线训练,让界面自动化更高效准确。
UI-S1: Advancing GUI Automation via Semi-online Reinforcement Learning
- 在离线数据中模拟在线交互,通过修复轨迹偏差提升性能。
- 在四个动态基准上超越70亿参数模型,最高提升23.8%。
- 适合需要高效部署且保持高精度的GUI自动化研究者。
图形用户界面(GUI)智能体通过强化学习在自动化复杂界面操作方面取得显著进展。然而,现有方法面临根本矛盾:离线强化学习虽能稳定训练,但缺乏任务级奖励信号,难以完成多步任务;在线强化学习虽能捕捉真实反馈,却受限于稀疏奖励和高昂部署成本。为此,我们提出半在线强化学习(Semi-online RL),在离线轨迹上模拟在线训练。每轮推演中,保留多轮对话中的原始模型输出,并通过修补模块自适应修正推演轨迹与专家轨迹间的偏差。为捕捉长期训练信号,该方法将折扣未来回报引入奖励计算,并联合优化步级与期级优势。我们进一步提出半在线性能(SOP)指标,更贴近真实在线表现,可作为实际评估的有效代理。实验表明,本方法在四个动态基准上实现7B模型的最先进性能,相较基础模型显著提升(如AndroidWorld +12.0%,AITW +23.8%),大幅缩小了离线训练效率与在线多轮推理能力之间的差距。代码已开源:https://github.com/X-PLUG/MobileAgent/tree/main/UI-S1。
原文摘要 · Abstract (English)
Graphical User Interface (GUI) agents have demonstrated remarkable progress in automating complex user interface interactions through reinforcement learning. However, current approaches face a fundamental dilemma: offline RL enables stable training on pre-collected trajectories, but struggles with multi-step task execution for lack of trajectory-level reward signals; online RL captures these signals through environment interaction, but suffers from sparse rewards and prohibitive deployment costs. To address it, we present Semi-online Reinforcement Learning, a novel paradigm that simulates online RL on offline trajectories. During each rollout process, we preserve the original model output within the multi-turn dialogue, where a Patch Module adaptively recovers the divergence between rollout and expert trajectories. To capture long-term training signals, Semi-online RL introduces discounted future returns into the reward computation and optimizes the policy with weighted step-level and episode-level advantages. We further introduce Semi-Online Performance (SOP), a metric that aligns better with true online performance, serving as a practical and effective proxy for real-world evaluation. Experiments show that ours Semi-online RL achieves SOTA performance among 7B models across four dynamic benchmarks, with significant gains over the base model (e.g., +12.0% on AndroidWorld, +23.8% on AITW), demonstrating significant progress in bridging the gap between offline training efficiency and online multi-turn reasoning. The code is available at https://github.com/X-PLUG/MobileAgent/tree/main/UI-S1.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。