用扩散模型生成假专家数据，提升强化学习模仿效果
SD2AIL: Adversarial Imitation Learning from Synthetic Demonstrations via Diffusion Models
- 用扩散模型生成伪专家演示数据，扩充真实数据
- 在Hopper任务中平均回报达3441，领先当前最佳方法89点
- 适合数据稀缺场景下的模仿学习，尤其适合仿真环境
对抗性模仿学习（AIL）通过专家示范推断奖励以指导策略优化。尽管更多示范通常能提升性能与稳定性，但在某些场景下获取真实示范仍具挑战。受扩散模型在数据生成中的成功启发，我们提出SD2AIL，利用扩散模型生成合成示范。首先，在判别器中使用扩散模型生成伪专家数据以扩充真实示范；为从大量（伪）专家示范中优先重放最有价值的数据，进一步引入优先级专家示范回放策略（PEDR）。在仿真任务上的实验表明，该方法有效且稳健。特别地，在Hopper任务中，平均回报达3441，优于当前最先进方法89点。代码将发布于https://github.com/positron-lpc/SD2AIL。
原文摘要 · Abstract (English)
Adversarial Imitation Learning (AIL) is a dominant framework in imitation learning that infers rewards from expert demonstrations to guide policy optimization. Although providing more expert demonstrations typically leads to improved performance and greater stability, collecting such demonstrations can be challenging in certain scenarios. Inspired by the success of diffusion models in data generation, we propose SD2AIL, which utilizes synthetic demonstrations via diffusion models. We first employ a diffusion model in the discriminator to generate synthetic demonstrations as pseudo-expert data that augment the expert demonstrations. To selectively replay the most valuable demonstrations from the large pool of (pseudo-) expert demonstrations, we further introduce a prioritized expert demonstration replay strategy (PEDR). The experimental results on simulation tasks demonstrate the effectiveness and robustness of our method. In particular, in the Hopper task, our method achieves an average return of 3441, surpassing the state-of-the-art method by 89. Our code will be available at https://github.com/positron-lpc/SD2AIL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。