用强化学习微调扩散策略，效果优于其他方法。
Diffusion Policy Policy Optimization
- 结合扩散模型与策略梯度，实现高效微调
- 在多个基准测试中性能和效率最优
- 适合机器人连续控制与真实场景部署
我们提出 Diffusion Policy Policy Optimization（DPPO），一种针对基于扩散的策略（如 Diffusion Policy）在连续控制与机器人学习任务中进行微调的算法框架，采用强化学习中的策略梯度（PG）方法。尽管以往认为策略梯度对扩散策略效率较低，但实验表明，DPPO 在常见基准上相比其他用于扩散策略的RL方法，以及与其他策略参数化形式的PG微调相比，均表现出最强的整体性能与效率。通过实验发现，DPPO 利用了强化学习微调与扩散参数化之间的独特协同效应，实现结构化且位于流形上的探索、稳定训练与强鲁棒性。我们进一步验证了其在多种现实场景中的优势，包括具有像素观测的仿真机器人任务，并实现了在长时序多阶段操作任务中，将仿真训练的策略零样本部署到真实机器人硬件的能力。
原文摘要 · Abstract (English)
We introduce Diffusion Policy Policy Optimization, DPPO, an algorithmic framework including best practices for fine-tuning diffusion-based policies (e.g. Diffusion Policy) in continuous control and robot learning tasks using the policy gradient (PG) method from reinforcement learning (RL). PG methods are ubiquitous in training RL policies with other policy parameterizations; nevertheless, they had been conjectured to be less efficient for diffusion-based policies. Surprisingly, we show that DPPO achieves the strongest overall performance and efficiency for fine-tuning in common benchmarks compared to other RL methods for diffusion-based policies and also compared to PG fine-tuning of other policy parameterizations. Through experimental investigation, we find that DPPO takes advantage of unique synergies between RL fine-tuning and the diffusion parameterization, leading to structured and on-manifold exploration, stable training, and strong policy robustness. We further demonstrate the strengths of DPPO in a range of realistic settings, including simulated robotic tasks with pixel observations, and via zero-shot deployment of simulation-trained policies on robot hardware in a long-horizon, multi-stage manipulation task. Website with code: diffusion-ppo.github.io
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。