让机器人在真实世界用离线数据高效学习,通过估计不确定性提升长时预测可靠性。
Uncertainty-Aware Robotic World Model Makes Offline Model-Based Reinforcement Learning Work on Real Robots
- 用不确定性感知的世界模型实现长时间滚动预测,误差传播更可控。
- 在真实四足和人形机器人上仅靠离线数据训练,性能超越基线方法。
- 适合需要低样本、高安全性的现实机器人控制场景,尤其适合已有历史数据的团队。
强化学习在机器人领域取得显著进展,但高性能算法通常高度任务依赖,难以复用已有数据。离线模型基于强化学习(MBRL)通过完全从现有数据集训练策略,提升了数据效率,但在长时滚动中易受误差累积和分布偏移影响。尽管现有方法在受控仿真中表现良好,但在真实机器人常见的噪声、偏差和部分可观测数据集上仍面临挑战。本文提出一个可落地的离线MBRL流程:RWM-U通过引入认知不确定性估计扩展自回归世界模型,实现时间一致的多步滚动预测,并有效传递长期不确定性。结合MOPO-PPO,将不确定性惩罚策略优化融入稳定且在线的PPO框架,适用于真实世界控制。我们在仿真与真实四足、人形机器人上评估了多种操作与行走任务,仅使用离线数据训练策略,结果持续优于无监督模型自由与不确定度忽略的模型基线。融合真实数据进行模型学习后,所获策略甚至超越仅在仿真中训练的在线模型自由基线。
原文摘要 · Abstract (English)
Reinforcement Learning (RL) has achieved impressive results in robotics, yet high-performing pipelines remain highly task-specific, with little reuse of prior data. Offline Model-based RL (MBRL) offers greater data efficiency by training policies entirely from existing datasets, but suffers from compounding errors and distribution shift in long-horizon rollouts. Although existing methods have shown success in controlled simulation benchmarks, robustly applying them to the noisy, biased, and partially observed datasets typical of real-world robotics remains challenging. We present a principled pipeline for making offline MBRL effective on physical robots. Our RWM-U extends autoregressive world models with epistemic uncertainty estimation, enabling temporally consistent multi-step rollouts with uncertainty effectively propagated over long horizons. We combine RWM-U with MOPO-PPO, which adapts uncertainty-penalized policy optimization to the stable, on-policy PPO framework for real-world control. We evaluate our approach on diverse manipulation and locomotion tasks in simulation and on real quadruped and humanoid, training policies entirely from offline datasets. The resulting policies consistently outperform model-free and uncertainty-unaware model-based baselines, and fusing real-world data in model learning further yields robust policies that surpass online model-free baselines trained solely in simulation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。