提出新型对抗性模仿学习框架,理论证明其高效且易于实现。
Adversarial Imitation Learning with General Function Approximation: Theoretical Analysis and Practical Algorithms
- 基于优化的在线学习框架,分离奖励与策略优化过程
- 两种方法均实现多项式专家样本复杂度,首次在通用函数逼近下保证效率
- 无需复杂设计,实测优于现有深度模仿学习方法
对抗性模仿学习(AIL)在模仿学习中表现突出,依赖神经网络近似取得实际成功。然而,现有理论分析多限于简化场景(如表格型、线性近似),且算法设计复杂,难以落地,导致理论与实践脱节。本文通过研究在线AIL在通用函数逼近下的理论基础,提出新的优化驱动型AIL(OPT-AIL)框架:对奖励学习进行在线优化,对策略学习采用乐观正则化优化。在此框架下,我们设计了无模型和有模型两种具体方法。理论分析表明,两者均能以多项式复杂度学习接近专家的策略。据我们所知,这是首个在通用函数逼近下被证明高效的AIL方法。实践中,OPT-AIL仅需近似优化两个目标,便于实现。实验显示其在多个挑战性任务中优于现有最先进的深度AIL方法。
原文摘要 · Abstract (English)
Adversarial imitation learning (AIL), a prominent approach in imitation learning, has achieved significant practical success powered by neural network approximation. However, existing theoretical analyses of AIL are primarily confined to simplified settings, such as tabular and linear function approximation, and involve complex algorithmic designs that impede practical implementation. This creates a substantial gap between theory and practice. This paper bridges this gap by exploring the theoretical underpinnings of online AIL with general function approximation. We introduce a novel framework called optimization-based AIL (OPT-AIL), which performs online optimization for reward learning coupled with optimism-regularized optimization for policy learning. Within this framework, we develop two concrete methods: model-free OPT-AIL and model-based OPT-AIL. Our theoretical analysis demonstrates that both variants achieve polynomial expert sample complexity and interaction complexity for learning near-expert policies. To the best of our knowledge, they represent the first provably efficient AIL methods under general function approximation. From a practical standpoint, OPT-AIL requires only the approximate optimization of two objectives, thereby facilitating practical implementation. Empirical studies demonstrate that OPT-AIL outperforms previous state-of-the-art deep AIL methods across several challenging tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。