arXiv:2603.22713cs.LG2026-03中稿 · ICML

提出新方法消除模仿学习中的误差累积问题,理论证明更优。

Non-Adversarial Imitation Learning Provably Free of Compounding Errors: The Value Flow Mechanism

  • 用贝尔曼约束实现值流传播,让未见状态也能学习。
  • 理论证明可完全消除误差累积,优于传统非对抗方法。
  • 适合追求理论保证的强化学习研究者使用。

对抗性模仿学习(AIL)通过减少行为克隆(BC)固有的误差累积问题,实现了高质量模仿,但其对抗优化常导致训练不稳定。一类非对抗的基于Q值的模仿学习方法(如IQ-Learn)被提出以解决此问题,普遍认为其通过在线环境交互优于BC。本文重新审视IQ-Learn,证明其实际退化为BC:其模仿差距下界与轨迹长度呈二次依赖,仍受误差累积影响。理论分析揭示在线交互无效的原因:IQ-Learn在演示未覆盖的状态上对所有动作统一抑制Q值,阻碍了超越演示的泛化。为此,我们提出基于贝尔曼约束的Dual Q-DM,关键在于驱动值流:通过环境动态,从演示状态向未访问状态传播Q值,实现泛化。我们证明Dual Q-DM等价于AIL,可在未访问状态恢复专家动作,从而消除误差累积。据我们所知,Dual Q-DM是首个理论上保证消除误差累积的非对抗式模仿学习方法。实验结果进一步验证了理论发现。

原文摘要 · Abstract (English)

Adversarial imitation learning (AIL) achieves high-quality imitation by mitigating compounding errors inherent to behavioral cloning (BC), yet its adversarial optimization frequently leads to training instability. A class of non-adversarial Q-based imitation learning (IL) methods, exemplified by IQ-Learn, has emerged to address this instability and is widely believed to outperform BC by leveraging online environment interactions. In this paper, we revisit IQ-Learn and prove that it in fact reduces to BC: it admits an imitation gap lower bound with quadratic dependence on the horizon and therefore remains susceptible to compounding errors. Our theoretical analysis reveals why online interactions fail to help: IQ-Learn uniformly suppresses Q-values for all actions at states not covered by demonstrations, preventing generalization beyond demonstrations. To address this fundamental limitation, we introduce Dual Q-DM, a new Q-based IL method built on Bellman constraints. Crucially, Bellman constraints drive value flow: Q-values propagate from demonstrated to unvisited states through environment dynamics, enabling generalization beyond demonstrations. We prove that Dual Q-DM is equivalent to AIL and can recover expert actions at unvisited states, thereby mitigating compounding errors. To the best of our knowledge, Dual Q-DM is the first non-adversarial IL method that is theoretically guaranteed to eliminate compounding errors. Experimental results further corroborate our theoretical findings.

模仿学习值流误差累积

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。