用扩散模型提升无动作示范的模仿学习稳定性与效果
Diffusion Imitation from Observation
- 将扩散模型用于生成状态转移，替代传统生成器
- 在多个连续控制任务中实现更优性能，尤其在导航与操控场景
- 无需动作标签，适合真实场景下专家行为学习
学习从观察（LfO）旨在仅通过状态示范来模仿专家行为，无需动作标签。现有对抗式模仿学习方法通过训练生成器策略产生难以被判别器区分的状态转移。然而这些方法对超参数敏感且训练不稳定。受扩散模型在生成建模中的成功启发，本文提出将扩散模型引入从观察的对抗式模仿学习框架。具体而言，利用扩散模型根据当前状态生成下一状态，以捕捉专家与代理的状态转移。随后，重新构建学习目标，使扩散模型作为二分类器进行训练，并为策略学习提供“真实性”奖励。所提出的框架DIFO（Diffusion Imitation from Observation）在多种连续控制任务中表现优异，涵盖导航、运动、操作及游戏等场景。
原文摘要 · Abstract (English)
Learning from observation (LfO) aims to imitate experts by learning from state-only demonstrations without requiring action labels. Existing adversarial imitation learning approaches learn a generator agent policy to produce state transitions that are indistinguishable to a discriminator that learns to classify agent and expert state transitions. Despite its simplicity in formulation, these methods are often sensitive to hyperparameters and brittle to train. Motivated by the recent success of diffusion models in generative modeling, we propose to integrate a diffusion model into the adversarial imitation learning from observation framework. Specifically, we employ a diffusion model to capture expert and agent transitions by generating the next state, given the current state. Then, we reformulate the learning objective to train the diffusion model as a binary classifier and use it to provide "realness" rewards for policy learning. Our proposed framework, Diffusion Imitation from Observation (DIFO), demonstrates superior performance in various continuous control domains, including navigation, locomotion, manipulation, and games. Project page: https://nturobotlearninglab.github.io/DIFO
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。