arXiv:2502.20168cs.ROcs.AI2025-02被引 16

用状态空间模型加速基于模型的强化学习,训练速度提升4倍。

Accelerating Model-Based Reinforcement Learning with State-Space World Models

  • 用状态空间模型并行化动态模型训练,突破计算瓶颈。
  • 在真实四轴飞行任务中,世界模型训练提速10倍,整体训练快4倍。
  • 适合需要高效训练的复杂机器人场景,尤其部分可观测环境。

强化学习是机器人学习的强大方法,但模型无关强化学习(MFRL)需大量环境交互才能学习有效控制策略,因训练更新噪声大且机器人系统动态高度非线性、传感器信号嘈杂。相比之下,基于模型的强化学习(MBRL)同时训练策略和世界模型,捕捉环境动态与奖励。世界模型可用于规划、数据收集或提供一阶策略梯度。利用世界模型显著提升样本效率。然而,同步训练世界模型会增加计算复杂度,导致训练时间过长,难以应用于复杂现实场景。本文提出一种基于状态空间模型(SSMs)的MBRL加速方法,通过并行化动态模型训练(主要计算瓶颈),大幅提升效率。此外,设计新架构在训练时向世界模型注入特权信息,特别适用于部分可观测环境。我们在多个涉及复杂动态的真实四轴飞行任务中评估该方法,涵盖全可观测与部分可观测情形。结果表明,世界模型训练时间减少达10倍,整体MBRL训练时间减少4倍,性能与顶尖MBRL方法相当,未牺牲样本效率与任务奖励。

原文摘要 · Abstract (English)

Reinforcement learning (RL) is a powerful approach for robot learning. However, model-free RL (MFRL) requires a large number of environment interactions to learn successful control policies. This is due to the noisy RL training updates and the complexity of robotic systems, which typically involve highly non-linear dynamics and noisy sensor signals. In contrast, model-based RL (MBRL) not only trains a policy but simultaneously learns a world model that captures the environment's dynamics and rewards. The world model can either be used for planning, for data collection, or to provide first-order policy gradients for training. Leveraging a world model significantly improves sample efficiency compared to model-free RL. However, training a world model alongside the policy increases the computational complexity, leading to longer training times that are often intractable for complex real-world scenarios. In this work, we propose a new method for accelerating model-based RL using state-space world models. Our approach leverages state-space models (SSMs) to parallelize the training of the dynamics model, which is typically the main computational bottleneck. Additionally, we propose an architecture that provides privileged information to the world model during training, which is particularly relevant for partially observable environments. We evaluate our method in several real-world agile quadrotor flight tasks, involving complex dynamics, for both fully and partially observable environments. We demonstrate a significant speedup, reducing the world model training time by up to 10 times, and the overall MBRL training time by up to 4 times. This benefit comes without compromising performance, as our method achieves similar sample efficiency and task rewards to state-of-the-art MBRL methods.

强化学习状态空间模型机器人控制加速训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。