用世界模型+预测控制，让扩散策略在离线到在线迁移中更高效稳定。
MODIP: Efficient Model-Based Optimization for Diffusion Policies
- 通过世界模型与预测控制生成高质量轨迹，作为扩散策略的监督目标。
- 仅需1步推理时间，比传统方法快3倍以上，在多个机器人任务上超越行为克隆。
- 适合需要稳定迁移的机器人学习场景，尤其适合资源受限的部署环境。
扩散策略（DPs）在机器人学习中表现出强大的表达能力，常与行为克隆（BC）结合使用。然而，其在直接强化学习（RL）微调方面仍面临挑战，因动作需通过多步去噪过程生成。本文提出MODIP框架，实现从离线到在线的扩散策略微调。MODIP不直接对DP施加RL，而是利用世界模型（WM）引导策略适应，并保持了行为克隆的简单性和稳定性。通过模型预测控制（MPC）在WM内生成高质量轨迹，作为微调DP的监督信号。为提升MPC效率，MODIP采用终端状态值而非依赖策略的状态-动作值，显著降低推理时间；同时使用独立于策略的TD目标训练价值函数，减少训练开销。在D4RL（MuJoCo、Kitchen）和RoboMimic任务上的实验表明，MODIP在性能上超越了行为克隆，且媲美或优于现有扩散策略强化学习微调方法，以及强基线如TD-MPC2。
原文摘要 · Abstract (English)
Diffusion policies (DPs) have emerged as expressive policy representations for robot learning, often used with imitation learning methods such as behavioral cloning (BC). However, while their success has largely been confined to BC, direct reinforcement learning (RL) fine-tuning remains challenging because actions are generated through a multi-step denoising process. In this work, we propose MODIP, a framework for the offline-to-online fine-tuning of DPs. Rather than directly applying RL to the DPs, MODIP leverages a world model (WM) to guide policy adaptation and keeps the simplicity and stability of BC. We utilize model predictive control (MPC) to generate high-quality trajectories within the WM, and use them as supervised targets for fine-tuning the DP. To make MPC planning efficient, MODIP uses a terminal state value instead of a policy-dependent state-action value, reducing inference time. Additionally, MODIP trains critics with policy-independent TD targets, reducing training time. Experiments on D4RL (MuJoCo, Kitchen) and RoboMimic tasks show that MODIP improves diffusion policies beyond BC, and is competitive with or outperforms diffusion policy RL fine-tuning methods and strong model-based baselines such as TD-MPC2.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。