通过构建世界模型生成高质量数据,提升机器人强化学习的样本效率与鲁棒性。
WOMBET: World Model-Based Experience Transfer for Robust and Sample-efficient Reinforcement Learning
- 用世界模型生成带不确定惩罚的规划数据,确保可靠性。
- 在连续控制任务上,比基线少40%样本达更高性能。
- 适合需要安全高效训练的机器人应用,如工业自动化。
机器人强化学习常受限于数据采集的成本与风险,促使从源任务向目标任务迁移经验。离线到在线强化学习利用已有数据,但通常假设数据集固定,未解决如何生成可靠转移数据的问题。本文提出基于世界模型的经验迁移框架WOMBET,联合生成并利用先验数据。WOMBET在源任务中学习世界模型,通过带不确定惩罚的规划生成离线数据,并筛选高回报、低认知不确定性的轨迹。随后在目标任务中使用自适应采样,在离线与在线数据间进行在线微调,实现从先验驱动初始化到任务特化适应的稳定过渡。我们证明了不确定惩罚目标提供了真实回报的下界,并推导出包含分布偏移与近似误差的有限样本误差分解。实验表明,WOMBET在连续控制基准测试中优于强基线,显著提升样本效率与最终性能,验证了联合优化数据生成与迁移的有效性。
原文摘要 · Abstract (English)
Reinforcement learning (RL) in robotics is often limited by the cost and risk of data collection, motivating experience transfer from a source task to a target task. Offline-to-online RL leverages prior data but typically assumes a given fixed dataset and does not address how to generate reliable data for transfer. We propose World Model-Based Experience Transfer (WOMBET), a framework that jointly generates and utilizes prior data. WOMBET learns a world model in the source task and generates offline data via uncertainty-penalized planning, followed by filtering trajectories with high return and low epistemic uncertainty. It then performs online fine-tuning in the target task using adaptive sampling between offline and online data, enabling a stable transition from prior-driven initialization to task-specific adaptation. We show that the uncertainty-penalized objective provides a lower bound on the true return and derive a finite-sample error decomposition capturing distribution mismatch and approximation error. Empirically, WOMBET improves sample efficiency and final performance over strong baselines on continuous control benchmarks, demonstrating the benefit of jointly optimizing data generation and transfer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。