用离线数据模拟在线反馈,提升长时序GUI导航的完成率
SOLAR-RL: Semi-Online Long-horizon Assignment Reinforcement Learning

- 从静态数据重构多种执行路径,识别首次失败点
- 通过目标对齐奖励设计,实现轨迹级质量评估
- 无需真实交互即可训练,适合低样本场景
随着多模态大语言模型(MLLM)的发展,GUI智能体正从静态交互演变为复杂导航。强化学习(RL)虽是训练动态GUI任务的有效范式,但面临困境:传统离线RL依赖静态步骤数据,忽略任务完成度与执行质量等全局轨迹语义;而在线RL虽能捕捉长期动态,却存在高交互成本和环境不稳定性问题。为此,我们提出SOLAR-RL(半在线长时序分配强化学习)。该框架不依赖昂贵的在线交互,而是将全局轨迹信息直接融入离线学习过程:从静态数据重构多样回放路径,利用每步有效性信号检测首次失败点,并基于目标对齐的奖励设计,为步骤级分配密集奖励以反映轨迹级执行质量,有效模拟在线反馈且无交互开销。大量实验表明,SOLAR-RL显著优于强基线,在长时序任务完成率与鲁棒性上均有提升,提供了一种高效的自主GUI导航解决方案。
原文摘要 · Abstract (English)
As Multimodal Large Language Models (MLLMs) mature, GUI agents are evolving from static interactions to complex navigation. While Reinforcement Learning (RL) has emerged as a promising paradigm for training MLLM agents on dynamic GUI tasks, its effective application faces a dilemma. Standard Offline RL often relies on static step-level data, neglecting global trajectory semantics such as task completion and execution quality. Conversely, Online RL captures the long-term dynamics but suffers from high interaction costs and potential environmental instability. To bridge this gap, we propose SOLAR-RL (Semi-Online Long-horizon Assignment Reinforcement Learning). Instead of relying solely on expensive online interactions, our framework integrates global trajectory insights directly into the offline learning process. Specifically, we reconstruct diverse rollout candidates from static data, detect the first failure point using per-step validity signals, and retroactively assign dense step-level rewards with target-aligned shaping to reflect trajectory-level execution quality, effectively simulating online feedback without interaction costs. Extensive experiments demonstrate that SOLAR-RL significantly improves long-horizon task completion rates and robustness compared to strong baselines, offering a sample-efficient solution for autonomous GUI navigation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。